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ABSTRACT 

One in a series of issue papers commissioned by the 
Council of Administrators of Special Education (CASE), this document 
presents a comparison of contemporary evaluation approaches for 
special education programs. The first section describes the two 
approaches to be compared: (1) traditional scientific inquiry which 
emphasizes quantitative methods; and (2) the naturalistic paradigm 
which emphasizes qualitative methods. The second section, written by 
Lester J. Horvath, is titled n A Quantitative Program Evaluation 
Approach to Evaluating Quality Special Education Programs, 91 and 
covers such areas as: the context of the problem (administrative 
challenge, reasons for program evaluation, contextual issues in 
special education, and training); assumptions regarding special 
education program evaluation; the recommended approach (including 
description, purpose and specific stages of evaluation); and other 
considerations. Section Three, written by Edith E. Beatty and titled 
"Qualitative Responsive Approaches to Evaluating Special Education 
Program Quality," addresses the local context; qualitative responsive 
evaluation; frameworks; and implementing qualitative responsive 
evaluations. The fourth section, written by Kenneth R. Clsen, is a 
comparative analysis of the approaches presented by Horvath and 
Beatty and discusses areas of agreement between the two approaches 
(stakeholder involvement, criteria for effective evaluations, 
emphasis on quality) and areas in which the methods differ (issues 
and concerns versus decisions, knowing design in advance, need for 
consensus). The final section outlines administrative implications of 
the findings and lists questions to help special education 
administrators in choosing the appropriate method for evaluating the 
quality of their special education program. (CB) 
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Section 

One: CASE and Program 

Evaluation 

In response to the public's call for quality 
and excellence in ecucation, special educators have come to realize that the previous 
emphasis on compliance monitoring is necessary but not a sufficient condition in 
program improvement efforts designed to achieve excellence. CASE has identified 
evaluation models and practices as a high priority for local special education ad- 
ministrators from the results of our research, conferences, and discussions with local 
leaders. The papers on program evaluation, teacher, and related staff evaluation 
have been distributed through the CASE Information and Dissemination service 
program. A third document highlighting evaluation practices currently being im- 
plemented around the United States was published jointly by CASE and the Office 
of Special Education Programs in the Department of Education. 

As part of its continuing commitment to provide information on the most 
recent developments in quality program evaluation, the CASE Executive Committee 
charged its Research and Special Projects Committee to continue in this focal area 
of interest with a document that compares contemporary program evaluation ap- 
proaches. Contrasting approaches have been a primary issue of discussion at con- 
ferences held by CASE over the past two years on the subject of program evaluation. 

Description of Approaches 

One approach is based upon traditional scientific inquiry and emphasizes 
quantitative methods. Horvath defines quantitative program evaluation as a planned 
attempt to measure the extent to which certain expectations about the program 
are being met. The second approach is based on a naturalistic paradigm and em- 
phasizes qualitative methods. Beatty defines qualitative program evaluation as an 
emergent process designed to discover the relevant values and expectations held 
by constitutiences who are served by the program. These two approaches can be 
reviewed along a number of continua. One continuum is that the quantitative or 
scientific inquiry approach requires a consensus on specific evaluation objectives 
prior to the conduct of the evaluation itself, while the naturalistic approach uses 
qualitative methods to discover the values or outcomes held by multiple stakeholders 
(persons with a stake in the special education program, including the full evalua- 
tion committee) in the educational setting. On another continuum, the methods 
and data collection instalments used in a quantitative approach are specified and 
linked to the specific outcomes during the evaluation activity. The methods used 
in a qualitative approach evolve throughout the process of evaluation Both ap- 
proaches, however, are credible, useful, feasible, and proper means to conduct quali- 
ty program evaluation. Given these distinctions, the choice of approach need not 
he "either/or," but dependent on the leader's perception of the purpose, values, and 
complexity of the program(s) to be evaluated within the local context. We collected 



two top special educators'views of these two processes to illustrate their respective 
advantages. 

Product Overview 

The two authors selected are Identified with these respective approaches. They 
are Lester J. Horvath representing quantitative and Edith E. Beatty representing 
qualitative. They were asked to draft their papers based upon five organizing 
questions: 

1. What in your view is the current context of special education and 
the need to consider program evaluation? 

2. What assumptions do you make about special education? 

3. How do your assumptions affect the conduct of program evaluation? 

4. What purpose do you hope to achieve in conducting an evaluation? 

5. How do you recommend a program evaluation be designed and 
carried out? 

These two papers are placed in Sections II and III of ;his document. 

When the papers were completed, they were sent out for CASE Research 
Committee and outside consultant reviews. The drafters, consultants, and Research 
Committee members read and diseased the papers on September 19, 1984, to 
determine similarities and differences in approaches and potential uses of these 
methodologies in local school program quality evaluation efforts. Two consultants 
were invited to critique the papers: Ken Olsen of the Mid-South Regional Resource 
Center in Lexington, Kentucky, and Constance Berquist of Evaluation Systems 
Design, Incorporated, Tallahasee, Florida. Ken Olsen was also asked to lead the 
discussion of the comparative strengths and weaknesses of each approach. His com- 
ments were of such quality, the Committee asked him to capture his suggestions 
along with suggestions of our Committee. These comments are placed in Section 
3 of this report. 

The Committee also developed a set of administrative implications that it 
hoped would assist colleaques in determining what approaches best fit the local 
context. These statements comprise the fourth and final section of this publication. 

We hope this document in the series, will give you a more com- 
plete picture of evaluation. This up-close look should help those crafting their own 
designs and implementation of school improvement efforts. 

L.C Bunvllo, Research Committee Chair 
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A Quantitative Pro- 
gram Evaluation 
Approach to 
Evaluating Quality 
Special Education 
Programs 

by Lester J. Horvath 
Associates in Professional 
Technologies, Inc. 
West Hartford, Connecticut 

The first paper Is written on the quantitative 
Approach. This scientific Inquiry approach requires a consensus on specific evalua- 
tion outcomes. The goals for all students should be considered as the starting point 
The methods and data collection Instruments used In a quantitative approach are 
specified and linked to the specific outcomes during the evaluation activity. In sup- 
port of this approach, some special education administrators said: 

• Our district wonts data on our program operation. They believe col- 
lected facts and figures are the best measure of a program's success. 

• We selected a quantitative model because it was simpler to administer 
and could be used right away. 

• This method required less time and was not as expensive as other 
models. 

• Questions were developed from the state of Massachusetts twelve 
goals for education and were adapted to our unique situation. The 
wording was altered to make the reading easier for respondents and 
an observation checklist was made for moderately handicapped 
students who could not complete a survey. 

• The quantitative method of evaluation was efficient and low In 
burden. 

Introduction 

From an administrative viewpoint, good program evaluation must provide 
practical information which can lead to program improvement and inform the ad- 
ministrator and other decision-makers about the program's progress. In order to 
do this, the program's processes (procedures and implementation) as well as the 
program's outcomes (products) must be studied within the context and realities of 
the specific situation. In special education this means a study of the quality of pro* 
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cesses such as Instruction, student assessment, IEP development, and outcomes 
such as student learning In bask skifls, appropriate student behaviors, and parental 
satisfaction with the program within the context of local expectations about the 
program. 

The Purpose of Thin Article 

This paper advocates a specific approach to special edcabon program evalua- 
tion for special education administrators, and specific strategies for employing the 
recommended approach* The attvoach b essentially quantitative, combining several 
of the most freqently employed design elements as they apply to special educa- 
tion. A major feature of the approach Is that various qualitative methodologies can 
be added to complement the basic model These alternative methodologies are 
described In the companion article within this Issues paper. 

The specific approach advocated In this paper relies on planning the evalua- 
tion activities early in the special education program evaluation process, based on 
the consensus of a committee of program stakeholders. The special education ad- 
ministrator includes parents, professionals, and representatives of all concerned 
groups In the formation of the stakeholder committee. This committee provides 
directions for and control of the evaluation throughout the process. The stakeholder 
committee provides structure through the identification of evaluation questions which 
define the scope of Items on evaluation Instruments. The Instruments yield quan- 
titative data, such as descriptive statistics for Interpretation and the development 
of a management plan. 

Context of the Problem 
The Challenge For Administrators 

Administrators of special education programs are responsible for the manage- 
ment and improvement of a very Important component of school district activities. 
This Importance comes from the size of the program, its complexity, and the ex- 
pectations of the people with a stake In the program. 

The size of special education is considerable In the United States alone, special 
education serves 4.4 million students at an annual cost of well over ten billion dollars. 
A special education administrator In a medium size local education agency (LEA) 
has approximately a one million dollar budget; the scope of large cities and In- 
termediate units Is of course, considerably higher. With respect to small LEAs, special 
education often requires disproportionate administrative attention. 

Perhaps no other education program matches special education for complexity 
of administration. Approximately 50 pages of federal regulations, pertinent state 
mandates, and a body of expanding case law constitute the legal backdrop. Com- 
pounding this procedural complexity is the necessity to provide for an extremely 
diverse population of students. There Is an abundance of state and federal reports 
to be submitted regarding the program. Professional practices change rapidly* and 
individual student progress is somewhat difficult to measure. Most special educa- 
tion students spend a substantial proportion of their day in regular education pro- 
grams, necessitating coordination. 
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The expectations of parents, students, staff, the community, and the funding 
agents are varied, but there is a common theme. A summary of that theme is that 
special education students must 

1. Be provided an education in accordance with the various mandates. 

2. Be provided educational experiences in accordance with sound pro- 
fessional practices (i.e., "good services"). 

3. Benefit from their educational experience in areas such as the basic 
skills, vocational skills, and social skills (citizenship, physical well-being, 
and other student outcomes are often included as expectations). 

The challenge for administrators, then, is to squarely address the expecta- 
tions of the various stakeholders through program evaluation, and make construc- 
tive use of the evaluation results. 

Reasons for Program Evaluation 

From the perspective of the special education administrator, the expectations 
of various stakeholders (including the administrator) give rise to four reasons for 
program evaluation: 

1. Program Improvement. The evaluation is used as a management tool 
for improving specific components of the LEA special education 
program. 

2. Policy Analyst The evaluation provides information to policy makers 
regarding programmatic issues that require resolution at a policy level. 

3. Accountability. The evaluation serves as an accountability ieport to 
an administrative, regulatory, oversight, or funding authority. 

4. Public Relations. The evaluation is used to provide information which 
is useful in a public relations or public information effort. 

In practice, special education administrators often conduct program evalua- 
tion for a combination of the four above reasons. Frequently there are primary and 
secondary reasons for evaluation. In some cases, additional use is made of evalua- 
tion asults post hoc. For example, the principal reason for conducting evaluation 
might be the use of the report as a management tool to identify specific program 
components (e.g., elementary LD resource rooms) that need attention with respect 
to specific program expectations (e.g., mathematics achievement). At a later date, 
the administrator might make use of the same data for policy analysis. The design 
could compare outcomes across resource rooms that employ different approaches 
to coordination with the regular education curriculum. 

If the evaluation were comprehensive, it could be used for an accountability 
reporter to the board of education in response to a request for information about 
the program's effectiveness. Finally, the administrator could include the report as 
part of the ongoing public relations effort for the program by disseminating the find- 
ings to various media. If the findings are positive, they would generate overall sup- 
port for the program. Negative findings can be used to support requests for addi- 
tional funding and/or politically difficult programmatic changes. 
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The following describes each of the four reasons for special education pro- 
gram evaluation with more specificity. 

Program Improvement. The use of program evaluation by ad- 
ministrators to systematically identify program areas that need administrative at- 
tention is an important aspect of administration itself. Administrators contlnously 
do ths kf. .d of evaluation informally and without benefit of systematic methodology 
and "rigor". 

Special education is an ongoing program of considerable magnitude, In need 
of systematic administrative attention. Recent evt luation literature has largely ig- 
nored the managerial perspective, This may be because, in the past, the bulk of 
evaluation was conducted on grant programs that could be terminated or con- 
tinued on the basis of the evaluation results. The reason for program evaluation 
in the competitive context of grant programs was a continuation or elimination 
of the program. Program improvement was a secondary issue at best. 

Fbr special education the termination of the program or even the termina- 
tion of subcomponents is a moot issue because the program and contiuum of 
services is mandated. Thus, program improvement, change, and restructuring are 
the focus of the administrator's decision making. 

It should be noted that this has important implications when planning a unit 
of analysis (sometimes called reporting unit) in the evaluation. From the ad- 
ministrative perspective it is important to ask for one's information in useable 
"chunks". The specific example here would be to structure the evaluation unit of 
analysis so that it corresponds with administrative planning units such as the LD 
elementary resource rooms. A poorly planned evaluation might report on too 
small a unit of analysis for the administrator (e.g., individual students) or too large 
a unit (e.g., combining the elementary and the high school levels). 

Policy Analysis. In the past, most policy analysis was done at the federal 
and state level through grants and contracts to private evaluation firms, univer- 
sities, and other research groups. As part of the decentralization of educational 
policy-making, policy analysis is increasingly being conducted at the local school 
district level. California's program evaluation system is the leading example of this 
trend within the field of special education. The arguments for allocating resources 
for policy analysis at the LEA level center around: 1) a better understanding of 
the contextual issues by LEA professionals; 2) improved technical expertise in 
many LEAs; and 3) an increased likelihood of use of the findings by other LEAs. 
Policy analysis evaluation has many similarities to applied research. As program 
evaluation is the child of research, this is the apple that fell the shortest distance 
from the tree. Specifically, policy analysis is often asked to "investigate" a propos- 
ed treatment, compare the treatment against a control, or compare several 
treatments. 

Some policy analysis problems center around lack of clarification of the ques- 
tion, rather than a need to compare proposed solutions. In such situations, the 
most valuable section of the policy analysis report is the section that identifies 
questions for further study. Qualitative methodologies are especially suitable for 
the identification of issues that lead to development of new hypotheses, due to 
the rich description of the program and the insightful interpretations provided. 

Accountability. This reason for program evaluation in special education 
stems from the need to provide comprehensive and impartial reports about the 
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program to the major formalized groups of stakeholders, These groups are school 
boards, local parent groups, the school superintendent, the administrative coun- 
cil (in some large districts), and the state department of education. (From the SEA 
perspective, these groups would include the state board of education, the state 
legislature and the U.S. Department of Education.) When an administrator is ask- 
ed to k port on program quality and effectiveness as an accountability mc hanism 
from one or more of these groups, the response should be related to ' * vpecta- 
tions of the group, as discussed above. In the past, administrators K " claimed 
that evaluation data on student progress is not possible. This response has be4*n 
considered inadequate by many school boards and parent qroups. 

When accountability is a significant purpose for conducting program evalua- 
tion, a self-evaluation approach directed by the special education administrator 
stands the likely risk of criticism on the grounds of lack of impartiality. Although 
this criticism can be countered to some degree if the evaluation uses materials 
and procedures developed from the outside (i.e., from the SEA or a private ven- 
dor), the administrator should be prepared for such criticism. Impartiality and 
avoidance of subjectivity are critical elements when conducting evaluation for the 
purpose of accountability. 

Public Relations. Special education administrators are very familiar with 
the need for public relations because such efforts have historically been necessary 
for program survival. Even today, after many years with mandates in place, some 
districts must defend their programs or face severe cuts that would force non- 
compliance with the mandates. Evaluation findings can complement the ad- 
ministrator's ongoing public relations efforts. 

The professional ethics of the special education administrator and the 
evaluator(s) are tested when evaluation results are used for public relations pur- 
poses. It is not enough to be passive with respect to the issue of ethics and evalua- 
tion; ttv? administrator should periodically remind all persons involved with the 
evaluation that an honest, ethical process is to be followed from conception to 
dissemination. 

If the evaluation is to be a self-evaluation by the LEA, the credibility issues 
noted above apply here and interact with the ethical issues. A self-evaluation us- 
ed for public relations purposes amidst a climate of distrust could "backfire", even 
if the evaluation and reporting were completely honest and even handed. 

It has been said that all evaluation has a political public relations effect, even 
when such effect is not aspired to (Cronbach, 1983). A practical guide for the 
administrator in preventing a public relations catastrophe is to keep the process 
as open as possible, and involve representative stakeholders throughout the pro- 
cess on an evaluation committee. It is no coincidence that these very stakeholders 
are also the core of political support for special education programs. Since the 
administrator cannot avoid the complicated areas of public relations and politics, 
it is suggested that a proactive, participatory approach be employed in this aspect 
of program administration. 

Contextual Issues in Special Education 

Evaluators from almost all schools of thought agree that the context (the set- 
ting of the program in the broadest sense) is a critical factor in the design and ex- 
ecution of good program evaluation. Special education poses some paradoxical 



twists for the uninitiated program evaluator who Is not familiar with this particular 
program. In some ways special education is very much like all of education, but 
in certain ways the program almost defies understanding from a program evalua- 
tion point of view. 

Similarities to Regular Education. Special education Is like all of 
education when it comes to the basic outcome expectations of the stakeholders. 
As mentioned above, persons with an Interest In the program expect students 
to profit In basic skills and the other areas commonly listed as goals for all educa- 
tion. With respect to process evaluation, an equivalent theme can also be 
developed for special education - sound professional practices should be employed 
in the delivery of services. 

When developing a quantitative evaluation system, the goals for ail students 
should be considered as the starting point. There is no need to spend dozens 
of professional days coming to the same conclusions that the local school board 
has formulated over the years. It is most revealing to note the similarities between 
special education and regular education when going through this process. This 
writer has sat through many committee meetings charged with developing "in- 
dicators of quality" (a preordinate evaluation step equivalent to goal setting in 
goal-based evaluation) only to have the belated discovery that the committee had 
essentially restated the same goals as the local board of education. Thus, It is 
recommended that the evaluation committee begin with the regular education 
goals. Each of these should be reviewed to determine their relevance, with the 
premise that special education students are entitled to regular education programs 
and services in accordance with their unique needs. Not only is this more effi- 
cient, it emphasizes the philosophical point that special education is not a separate 
entity from regular education. Special education usuafly has some additional pro- 
gram goals and objectives, the bulk of which are process oriented. 

Contrasts with Regular Education. Some contextual issues are very 
different from regular education. Special education has many activities that are 
quasi-evaluation in nature or whose terminology is confusing to the evaluator. 
For example, the individualized education program process has an annual review 
for each student; this is program evaluation with the student as the unit of analysis. 
There is, of course, student assessment and evaluaMon, and comprehensive stu- 
dent evaluation every three years as required by section 300a.534 of the federal 
regulations. Some states employ a program evaluation mechanism for those pro- 
gram components funded by Part B flow-through funds. Almost all grants award- 
ed through the twenty percent Part B state share funds require a special evalua- 
tion. Public Law 89-313 programs have an evaluation requirement. Most school 
districts employ some form of teacher evaluation which covers special education 
staff. Finally, there is the process of compliance monitoring. 

Compliance monitoring by the state education agency is a form of program 
evaluation that is limited to a portion of the process issues that can and should 
be addressed. Basic compliance monitoring addresses the implementation of man- 
dated processes, but it does not address the quality of such implementation. For 
example, compliance monitoring checks to see that each student's IEP contains 
objectives. It does not check the degree to which the objectives, taken as a whole, 
define a good program for the student for the coming year. 

Testing of students in special education is an important contextual issue. 

16 



Program ©valuators who do not understand the purposes for and the limitations 
of student assessment in special education often attempt quantitative evaluation 
using studc.it testing as the dependent variable, This has not proved fruitful for 
several reasons. The variance within the special education population is so great 
that It masks accurate measurement. The tests themselves are often not validated 
for application to this heterogeneous group, as contrasted *o the assumptions made 
for Chapter 1 program evaluation (e.g., using normal curve equivalence scores). 

Special education student prevalence varies widely from LEA to LEA, In 
some cases learning disabilities are Identified at a two percent prevalence rate, 
while some communities Identify well over ten percent of their students as learn- 
ing disabled. This variability In prevalence rates prohibits comparisons and tradi- 
tional pre-post designs that require objective criteria for program success (e.g., 
nine months again In mathematics per year), The districts with a higher prevalence 
rate would tend to show greater testing gains with their LD populations because 
the population would, as a whole, have less severe problems. 

Training 

It was noted earlier that there Is considerable turmoil In the program evalua- 
tion literature regarding methodology. There are conflicting ideas regarding the fun- 
damentals of research and evaluation design. Although it appears that the debate 
will not be readily resolved, practicing program evaluators can and do benefit from 
the exchange of ideas. Strsngths and weaknesses of alternative designs can be con- 
sidered in terms of actual apr^cations for special education program evaluation. 
Of course, special education administrators do not have the time to be fully im- 
mersed in current design isstu.:.. Although ^ !ministrators have shown considerable 
recent interest In program evaluafr i, it is not reasonable to expect a full-time ad- 
ministrator to have detailed knowledge of the evaluation profession. (Only the legal 
prrfession is entitled to that much of an administrator's time.) 

With the developing interest in program evaluation and the recent changes 
in the field of evaluation, special education administrators will need some sort of 
training. Even those special education administrators not planning to conduct the 
studies themselves need to keep abreast of current concepts in research and evalua- 
tion. Most special f ducation administration programs do not require a course devoted 
to program evaluation. One can assume that an administrator who took the stan- 
dard fare of related courses (eg., research design, tests and measurements, statistics, 
experimental psychology) before the mid-l97(ys was exposed to what was then the 
state of the art - traditional, quantitative experimental design. During the interven- 
ing period, quantitative designs have improved and evolved, and qualitative designs 
havs emerged. The "state of the arf has changed. 

As one should know the reason for evaluation before designing the evalua- 
tion, one should know the reason for training before selecting inservice. In a 
workshop of a few days or less, an administrator can learn the overall concepts 
and become an intelligent consumer of evaluation services. It is not possible to learn 
how to evaluate one's program after a few days of seminar. There have been 
numerous attempts at turning administrators into evaluators with workshops over 
the last ten years; the lack of administrators conducting evaluation is testimony to 
the failure of that approach. 



Prepared material* and ongoing support seem to be the two key Ingredients 
In support of successful training for selfevaluation of programs by administrators. 
The SEA can provide considerable assistance In th? area, 

Terminology 

The special education administrator will notice that program evaluation ter- 
minology is misleading In some areas. Many terms are value laden, The terms quan- 
titative and qualitative, for example, both describe methods that attempt to measure 
quality. The distinction rationallsitic/naturallstic Is often Interchanged with quan- 
titative/qualitative. To appreciate the subtle distinctions in terminology that are at 
the center of this vigorous professional debate, the reader needs to consider the 
philosophical assumptions behind the various approaches. 

Evaluation professionals have been vigorously contrasting and arguing the 
merits of methodologies on philosophical and practical levels. There is a considerable 
debate as to whether the two approaches can be combined in a given study, due 
to their philosophical contrasts. A review of literature for the last ten years shows 
increasing interest In qualitative methodologies in research and evaluation. 

There is a discrepancy between the professional literature and current prac- 
tice In terms of methodology use. Miles and Huberman (1984) state that few working 
researchers are not blending the two approaches. In practice, the vast majority of 
program evaluation (across program specialties) is conducted using quantitative 
designs combined with some qualitative methods. In general, practicing program 
evaluators are not purist regarding the methodologies they employ; rather, they 
tailor their methods as they approach each situation. This is done to capitalize on 
the positive aspects of the various specific methodologies. 

It has been suggested (Lynch, 1983) that the terms quantitative and qualitative 
describe types of data rather than methods of evaluation, and that methods fall 
on a continuum with naturalistic inquiry at one end and experimental inquiry at 
the other. This type of framework would allow for the definition of a type of evalua- 
tion that is naturalistic (i.e., the evaluator has not manipulated the pro- 
gram/treat ants) and quantitative (i.e., the .valuator collects numeric data rather 
than verbal desc ptions). Using these definitions, a great deal of present evalua- 
tion is naturalisiti- inquiry with quantitative data. 

Until the field of evaluation agrees on a common terminology, it is suggested 
that the special education administrator focus on the basics of the evaluation pro- 
cess rather than the labels for approaches. 

State of the Art 

Extended Monitoring Systems. Many LEAs have experienced, either 
as a self-evaluation or as an evaluation conducted by the SEA, an activity that 
extends compliance monitoring beyond mandated issues and into additional areas. 
These efforts are not comprehensive evaluation, but they have filled «i need dur- 
ing a period of special education history when LEAs struggled with makng the 
new special education mandates work. 

Such systems typically mix quantitative and qualitative methodologies. For 
example, the Michigan SEA monitoring model (Michigan State Boerd of Educa- 
tion, 1981) includes items that inquire as to whether or not special education map- 
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date* have improved the education of handicapped children, A negative answer 
from some proportion of respondents would not prove noncompliance, This is a 
policy evaluation question, not a question on compliance determination, The 
Michigan manual also includes questions related to process evaluation that go 
beyond compliance determination. Many states have two compliance systems, 
basic and extended. In New Jersey there is a basic monitoring system and a Level 
II system, the latter being an extended monitoring system that formally recognizes 
the distinction between basic compliance monitoring and their extended monitoring 
systems*. 

A review of several of these extended monitoring systems reveals signifi- 
cant problems. The focus of the evaluation tends to be compliance Issues and 
the quality of the compliance process. This is a different focus from an evaluation 
that measures the quality of the educational process. Extended compliance systems 
generally do not address outcomes. Such systems also tend to be extremely labor 
Intensive. The system could also cause problems in the relationships between an 
SEA and LEAs If the distinction between mandates and subjective issues becomes 
blurred. It should be noted that only a few SEAs have the resources to assure 
consistent, quality evaluations using an extended monitoring system. 

Two Statewide Efforts. Approximately ten years ago California and 
Massachusetts required that their LEAs conduct program evaluation. The early 
efforts In these two states met with limited success, even though workshops we*e 
held to assist LEAs. Despite the workshops, LEA adminstratorsdid not know how 
to actually conduct evaluation. To compound the difficulties, compliance issues 
were often contused with the purpose of program evaluation, and the evaluation 
report was often viewed as an additional paperwork exercise associated with the 
LEA Plan. When both SEAs provided a manual for program evaluation along 
with ongoing support, useful program evaluation becomes a reality for a large 
number of LEAs (Massachusetts Department of Education MDE, 1981; Califor- 
nia State Department of Education, 1982). At this point in time, it can generally 
be said that LEAs in these two states have conducted successful evaluation through 
more than one cycle. 

Assumptions Regarding 
Special Education Program Evaluation 

General Assumptions 

The following comprise the writer's general assumptions about special educa- 
tion program evaluation: 

1. Special education program evaluation is possible. 

2. Initial efforts at program evaluation should be modest in terms of work 
and complexity. 

3. It is better to do a modest program evaluation then to do no pro- 
gram evaluation at all. 

4. The focus of the evaluation will become the focus for the program 
itself (this is called "goal L ropism u ). 



5, A specific evaluation design should depend on the rea«on for the 
evaluation, Ihe specific local context, and who will conduct the 
evaluation 

6. Evaluation should address both process <tncl outcomes. 

7. Evaluation should be an honest and open process. 

8, The people closest to the program are excellent sources of Informa- 
tion about the program's effectiveness. 

Adiom Ntrative Assumptions 

T he follow ' ssumptlons focus on the administrative perspective: 
1 Progrr i \ tatlon must not be confused with teacher or staff evalua- 
tion, dud this must be made clear at the beginning of the evaluation. 

Cs Most spedal education adminlstrr ors do not have the time to manage 
a program evaluation unless prepared materials/methods and con- 
sultatlve support are provided. 

3. The evaluation design and reporting format should not be so com- 
plex or esoteric as to "lose" the Intended audience. 

4. It Is critical to involve the stakeholders from the beginning of the 
evaluation, through a vehicle such as an evaluation task force. 

5. The evaluation should burden the staff and other respondents as lit- 
tle as possible; wherever possible, existing data should be used. 

6. The evaluation process should be a positive experience for the special 
education department staff; no special education staff member should 
be left out of the process. 

7. It is important t > d« 'ermine the involvement of regular educators In 
the evaluation process; in most LEAs all principals and some regular 
education teachers should be involved in any comprehensive 
evaluation. 

8. Ownership of the evaluation by parents and staff will assist greatly 
in effecting improvements recommended in the report. 

9. Follow-up of a program evaluation requires at least as much work 
as development of the report. 

The Recommended Approach 
Description and Purpose 

The approach recommended in this section stems from the contextual issues 
and the assumptions described above. It has evolved from reviewing numerous 
special education evaluation works over recent years from the perspective of 
evaluator and evaluation materials developer. 

Overview of Stages. The approach consists of eight stages, which 
organize the process and allow for the incorporation of various theoretical models 
at critical points in the process. The eight stages follow: 

1. Defining the LEA Context 
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2. Developing the Dt?»ign and Instrumentation 

3. Collecting lha Evaluation Information 

4. Analyzing the Data 

5. Interpreting the Dnta 

6. Writing and Presenting the Evaluation Rupoit 

7. Applying the Management Plan 

8. Conducting Follow-up Evaluation 

Theoretical Base, As stated earlier, practicing evaluators generally use 
a blend of methods In any given application, This ma*es It ospeclolly difficult to 
describe an applied model using theoretical labels such as "Discrepancy Evalua- 
tion" or "Goal-Free Evaluation: Rather than beg the Issue, however , a descrip- 
tion of the methodological and theoretical bases follows. 

Instead of structuring the evaluation around a specific evaluation 
methodology, set of program goals, or Instrumentation, it Is suggested that the 
process of evaluation itself be the structure. This allows for the Incorporation of 
individual methods that best suit the needs of the individual LEA at eich point. 
A comparable structure, which can accommodate various method* , was employed 
by the Center for the Study of Evaluation in designing their Program Evaluation 
Kit (Morris and Fltz-Glbbon, 1978). This concept of evaluation -based organiza- 
tion Is similar to Stufflebeam's (1983) Context Input Process Product (CIPP) model, 
because It emphasizes the administrative perspective and focuses on the deci- 
sions to be made. Gable (1982) described a special education program evalua- 
tion process through a series of stages based partially on an adaptation of the 
CIPP decisions. 

The recommended approach should be categorized as in the quantitative 
school because the evaluation design is planned in the early phases of the evalua- 
tion, and the focus of information collection is numerical data from a variety of 
instruments. Regarding the quantitative/qualitative definitions, it should be noted 
that some of the instruments (e.g., interviews, case study review sheets) used in 
a typical application of the recommended approach are often considered 
qualitative. The main distinction in defining the character of the recommended 
approach is its dellberateness with respect to direction. That Is, the evaluation is 
purposeful in terms of investigating the extent to which certain expectations about 
the program are being met. It is recommended that these expectations and the 
direction be set by a broad-based committee representing the various stakeholders 
during the first phase, as part of defining the LEA evaluation context. 

Specific Stages in the Evaluation 

The following paragraphs describe the eight stages of the evaluation process. 

Defining the LEA Context This first phase of the evaluation involves 
discussing the purpose of the evaluation, the resources, program structure, and 
other aspects of context evaluation (Stufflebeam, 1983) in a broad-based com* 
mittee format. Special education administrators are by necessity experienced in 
sharing authority with consumers and staff through the team process. In this set- 
ting the open process achieves what House (1980) termed "democratizing evalua- 
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Hon," Using Ihli type of open proce&t to establish evaluation quest'ons allow* the 
\mt of two moduli, stakeholder and goalbased evaluation, as advocated within 
th«3 context of special education program evaluation by Ando (1984), 

A working subcommittee needi to be assigned tor specific tasks during this 
phase. After the subcommittee It assigned, It can begin to do the ground work 
for the evaluation. Program objectives regarding both process and product (out- 
comes) should be drafted In consideration of the programmatic contort of the 
LEA, Subcommittee members should Interview a broad spectrum of stakeholders 
in the formulation of these program objectives, It is suggested that the subcom- 
mittee conduct these Interviews uilng a combination of structured question* and 
open ended questions. The structured questions should be derived from existing 
LEA document! that state educational goals for regular education and for special 
education. For example, the hoard of education may have passed a resolution 
stating that all students age 16 and above should be provided an opportunity for 
vocational skill training. The interview guide in this example would then pose the 
question, To what extent do you feel that availability of vocational skill program- 
ming ts an Important program objective for special education in our schools?* Note 
that this example Is a process evah» a M ™ - bjectlve in that it addresses the delivery 
of a program rather than a stud* ut *»me. If the board resolution had been 
that students should acquire vocational skills, the objective would be considered 
a product (outcome) objective, An Important discussion among subcommittee 
members as they develop an interview guide is to determine whether or not an 
issue Is process, product, or both. In this example, the subcommittee could decide 
to expand the Issue to address product as well as process. The Implementation 
of this specific issue is also a po^nt to be determined. It should be stressed that 
the subcommittee has the latitude to apply common sense and the Implicit con- 
cerns of the stakeholders in the development of the interview guide and in the 
Interpretation of the results. This should not be viewed as a lock step process, 
beyond the control of the evaluators. A "pure" and traditional goal based evalua- 
tion does not allow for flexibility at this point in the process. The recommended 
approach in this article stresses the concerns of the stakeholders and is respon- 
sive to the context of the program. 

After the subcommittee summarizes the results of the interviews, the sum- 
mary should be presented to the evaluation committee for revision and adop- 
tion. A product of this stage of the process is a list of program process and pro- 
gram product evaluation questions as described by Oben (1979) and Gable (1982), 
or as discussed in the Massachusetts system (MDE, 1981). 

Developing the Design and Instrumentation. The next phase of 
the evaluation consists of the actual design. Put simply, the first order of work 
here consists of determining how each evaluation question should be answered. 
The evaluator's judgment is called for when determining how many instruments 
(sources of information) are sufficient to address each question. Specific steps 
in this phase include the need to: 

1. Determine proper evaluation approaches for each of the program 
objectives, and reach committee consensus on the suitability of each 
approach. 
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2, Develop the specific instrumentation, assignment*, #imJ induction* 
(or each of (he evaluation approaches 

3. Develop design median cs, such as the unit nf analyst* and lh* wimpl 
ing plan 

4 h!ot test the instrumentation and make neceisarv revision* 

5 Proofread and print sufficient quantities of the instrumentation 

Collecting tht> Evaluation Information, Although staff and parents 
may have been told about the evaluation in a general way when the precast was 
initiated, it ii a good idea to notify them at this point and explain that this U a 
program evaluation (not itaff evaluation) and that th^ir cooperation with surveys, 
interviews, rtc. will be greatly Appreciated Some IE As do thto notification through 
an article in their regular newsletter to the community, while others send a ipecial 
mailing, 

The recommended approach described h th's article often Involves Inter- 
views and folder reviews Interviewers need to be trained on the interview pro- 
tocol, and case folder reviewers need to be trained on actual case material to In- 
crease reliability and agree on a format for any qualitative data recording, 

The spe<if'c instruments need to be distributed according to the sampling 
plan, and a system for following up return of completed instruments should be 
established. An important point in collecting information is to obtain the max* 
imum rate of return for each instrument. Issues of access and confidentiality of 
data need to be addressed. 

Although the recommended approach is primarily quantitative, a good deal 
of anecdotal Information, planned and incidental, is usually collected during the 
process. In some cases this information is quite useful, but in other cases it is 
unrelated to the purpose for evaluation. The person(s) in charge of such data 
have three responsibilities: logging the data according to some logical schema; 
and, presenting periodic updates on the information to the evaluation committee 
to see if areas for additional systematic data collection are warranted. 

The final steps in collecting the data are also important. Nonrespondents 
should again be asked to provide the information to increase rates of return, All 
files and materials must be returned to their exact place of origin. People who 
conducted interviews, reviewed cases, etc. should be promptly "debriefed" so that 
their impressions can be obtained on common themes, incidental findings, and 
any identified limitations regarding the data ' hat was collected. 

Analyzing the Data. The data analysis techniques to be employed vary 
from evaluation to evaluation, but some common points can be put forth as sug- 
gested practice. In general, however, descriptive statistics should comprise the bulk 
of the analysis. Terms such as "percentage satisfactory* are especially useful to 
the reader of the report. The "percent** is one of the most widely understood 
concepts. 

If the purpose of the evaluation was to statistically compare groups, and 
the instrumentation and sampling were designed in a manner to support such 
analysis, the proper difference test would be performed at this stage. In general, 
such traditional quantitative analysis is not helpful to the evaluation. The excep- 
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iiuni are usually latger policy studies* where generalisation pf piinapl^n mm% 
I F.Ai ar settings is part of tht* evaluation purpose Undue emphasi* on statistic* 
is usually somewhat harmful for jir ogram improvement studies, because the focus 
too easily become the detail of the numbers rather than how to Improve the pro 
gram Tin* d^ba^ about practical iiynifiearKe vs statistical ilynificaned applies 
to this situation tor example. increasing the sample v * affects statistical analyses 
In a manner which may not translate to an increase in practical significance, 

Computer! have been increasingly Helpful In taking the burden out of data 
analysis The common mainframe statistical packages such as Statistical Analysis 
System (SAS) and Statical Package for the Social Sciences (SPSS) provide quick 
and dependable descriptive and inferential statistics Evaluation using the retain 
mended approach has been conducted using a microcomputer for all data analysis. 
It is suggested that a small sample of computations be conducted by hand, to 
ensure that the computer is doing the intended calculations 

Interpreting the Data. The commit*** as a whole should be involved 
in data interpretation, Discussion of the results at this stage yields a list of major 
findings, When important evaluation questions remain unresolved due to con 
flicting data, additional Information should be collected as deemed necessary by 
the committee, The overall focus of data interrelation ttwuM be the process and 
product findings as they pertain to the purpose of the evaluation, 

Writing and Presenting tha Evaluation Report Writing the report 
is an extention of the Interpretation activity described above. The focus of the 
committee at this stage turns from identifying the findings to developing reconv 
mendattons and plans for action. 

The reader of the report should be provided an overview of the logical flow 
of evaluation processes from the first phase of the evaluation (defining the LEA 
context) through the development of recommendations and management plan, 
Clarity of purpose and understanding the perspectives of the audiences ere im- 
portant guidelines in writing. Graphic displays, executive summaries, and a full 
final report are the essential products. 

An important part of the final report is the management plan. Traditionally, 
evaluation reports have concluded with a recommendations section. It is suggested 
that this be taken a step further, with a plan to implement the recommendations. 
A limited number of important findings should be developed into plans with prac- 
tical approaches, timeline s, persons responsible, and resouces needed. A well- 
stated management plan can assist in securing resources and making politically 
difficult changes. 

Applying the Management Plan. Even though the most pervasive 
debate in evaluation concerns methodology, the biggest failure of evaluation has 
been administrative foDow up for program improvement. Recent examples, 
however, have demonstrated success in applying program evaluation. 

A well-written management plan can serve as a roadmap for the special 
education administrator. Placing an emphasis on the major points in department 
meetings, inservice, and written communications can assist making positive 
changes. The stakeholder group representatives (e.g., principals and parents) who 
were involved in the evaluation can help considerably in this stage. 

Conducting Follow-Up Evaluation. This stage of the recommended 
approach is part of the application of the management plan, but goes a step beyond 
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in thai it \% the beginning of the neat cycle Then are several advantage* t« using 
mm* of th# same evaluation questions (and instrument items) in repeated evalua 
linn* far some of the basic pfapam espectatiani such at student skill acquiti 
tinn» LEA baseline data have been established The success of the point* in the 
management plan can be determined, with new approaches and/** additional 
resource* allocated to those areas that continue to need attention The bwden 
of instrument development itself is also reduced 

The focus of each successive program evaluation should be evolutionary, 
keeping important program goals ami stakeholder concern* from past evaluations, 
dropping issues no longer important, and adding new pioatam priorities ite* 
ween evaluations, the LEA should devote evergy to Ihe follow up of manag >- 
m«nt plans Thus, program evaluation i* a dynamic and integral part of special 
education administration 

Other Consideration! 

The following important points deserve the special education administrate* 
attention when planning for program evaluation 

Validity Issut s 

In fcentftal, the source of validity for each evaluation question stems from the 
fact that multiple sources of information are employed to address the question 

This definition of validity is sometimes difficult to understand by persons who 
have a beck^ound in traditional tests and measurements Such persons have been 
trained that extensive validity and reliability statistical measures must be obtained 
on all Instrumentation before use. This to not applicable for evaluation instruments 
that have been individually designed for application in a specific context. Rather 
than attempt a series of statistical measures, a far more meaningful approach to 
validity is achieved by comparing Information from multiple sources within the 
evaluation, for example, if the student records checklist agrees with teacher surveys 
and parent interviews regarding mathematics achievement by elementary students 
served In the resource rooms, a high degree of confidence can be placed In the 
findings. 

Reducing Burden on Staff and Intrusion on 
Programs 

Too often, an otherwise sound activity Is not worth pursing because of staff 
burden and Intrusion on operating programs. Evaluation can definitely fall Into this 
trap. The evaluator and the evaluation committee have a great deal of work to 
do, and that work must be done with a high degree of rigor. The respondents such 
as teachers, administrators, parents, and related services staff In many LEAs have 
been deluged with surveys and meetings for a host of projects over recent years. 
To the extent that past experiences with such interruptions have taken place, indif- 
ference or even resentment on the part of the respondent may occur. Thus, evalua- 
tion activities have the burden of proof regarding their value. 

The first steps in conducting evaluation are extremely important. It is assum- 
ed that outside assistance in design and Instrumentation will be obtained. If possi- 
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Ne A' evaluation «»«pert faroilww wan tfmrt *b#*nm and tto setowd model 
stouta to consulted tetpwdinfi technical aspect!, fto au*> el ito sutoianaal mw 
comnutmeni required few ito *v#»w4*ri ito <t**iAl education edmeu**** shouM 
pUn to ml aside approwmately five munihs A stoulat f emnutment el mm to tto 
f**to ol waff and parent* rorofiroing ito nvatefton eammwee *touM ate to 
secured 

toll- evaluation without (hi tolp ol prepaid tiNMena!* i» to mote button 
some than w«h ito use of prepared m*i«>»wi» m cone actum *«h an estotnel 
evaluate One r#**on to Ito pttot test of instrument* » to ditermm* ito length 
oi time to respondents and to tensatet tot* lector wton revtong instrvmentetton 
It is to easier to create todmww mufcument* Iton good, clean instrument* wtoh 
quickly obtain Ito necessary toteima«©n to d ih« reason thai s® many 
restwndents resent suiveys is past wperience with surveys ihM appeared to to pondy 
designed test drafts Tto special education administrator should retort tin to et 
tor attitude and eapertencet regarding survey* (and requests tm mtomatton and 
interview*) during tto instrument deve lopment and tamptog plan stage* ©I tto 
project 

Evaluation Coats. 

Closely related to burton, the issue el cost is wry much a factor w setecttog 
a specific evaluatton rtrat*>av In tto case ©f outside valuators, ito cost atone 
become i tto de* img (actor for soma LEA* Sal evaluation can be conduced tor 
considerably fens, although equivalent costs ate essential pato cd tote* by 
tto (pedal education administrator and stag 

Wton program evaluation tort became standard practice w » » <•»•! pro- 
grams, a common tunderd wet to set askte five percent ol grant ■ , (or pro- 
gram evaluation purpose* That eandard do«a not apply to ongon * Uduca- 
tton proy»m» toc*uw »uch progimi do nol o>p*nd oo powow ) mute 

to ttofc annual contmuatlon Tto agpagMa co« ol a Ihw percent Hwauid 
to ttaggartno. Ahbough tpadal education program evttoedon H to ne »«• en 
ducted partly lor reetons ol accountobtliy. It It more often prtodpeBy UMd at a 
maneaement tool by tto ipedal educatton edmtototrator Uitog tto reaeon at tto 
focui. coit tMective evakiatlon c*n to defined as eveluetton ttot Improve* the pto- 
8r»m. m a quakty tenae. more than tto expendttuttotttotwrwtovbu^ tove 
produced through direct »trvk« For tuampk would • S 12.000 program rvahi«- 
tton <rv»ry thr«r ywan Improv* th« program moro lhan $4,000 worth erf eddlttond 
staff for each ol thou three ytara? (Tor clarity, this example Ignores Interest and 
Inflation ) This general rult Is useful In measuring tto relative cost-effectiveness ol 
any evaluation. 

Frequency of Evaluation 

There Is an obvious connection bttwaen tto above rwo sections dealing with 
burden and cost, and the kuu« ol frequency ol evaluation. For these reason* many 
LEAs decide on comprehensive evaluation on a three- to five-year cycle. During 
the yean between evaluations, these LEAs devote adminisfrative attention to fo&jw- 
ing though with tto program improvements specified in their evaluation report's 
management plan. 
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Some LEAs choose to evaluate only a portion of their program each year. 
This approach in effect addresses a cluster of units of analysis and provides relatively 
easier differentiation of approaches and evaluation questions among program com- 
ponents. To return to the assumption that some evaluation is better than no evalua- 
tion, evaluation nf program components may be the best decision for some LEAs. 

Ensurtag Evaluation Use and Follow-up 

It cannot b*5 emphasized strongly enough that evaluation should not be con- 
ducted if the. report will merely sit on the shelf. If the program evaluation produces 
a management report section with specific responsibilities for various staff members 
and reasonable dates by which certain actions must be taken, the use of the evalua- 
tion Is far more certain. 

A sense of ownership of the evaluation report by staff, parents, and other 
stakeholders is a contributing factor to evaluation use. If the special education ad- 
ministrator dominates the evaluation at the early stages, broad-based ownership 
is not likely to occur. It is suggested that the administrator foster a healthy diversity 
of opinion within the evaluation committee, and yiekJ in deference to persons closer 
to some of the issues during the committee's work. 

A Partnership with Parents and Staff 

A natual continuation of the discussion of ownership is the development of 
a common purpose among the stakeholders in a program. As an anecdotal exam- 
pale, the parents in one LEA were very much included in the management plan 
as a follow-up to the program evaluation. Some of the management plan actions 
dealt with funding speakers for the local parent group. Other actions of particular 
interest to parents received prompt attention by the school superintendent. From 
a long-term perspectiv e, an improved partnership between parents and the school 
is probably even more important than the specific management plan actions. 

Special education has made great advances in structuring parent-school part- 
nerships within various school activities. It is natural that such partnerships will be 
built into the program evaluation process as well. Special education administrators 
know well that this partnership is more than an end in itself. It is a cornerstone 
of the legitimate political process of program improvement. 

Evaluation Standi ds 

Accepted standards, such as those put forth by The Joint Committee on Stan- 
dards for Educational Evaluation (1981) should be followed in the implementation 
of the evaluation. These standards can be viewed to ensure that the evaluation 
is ui ll, feasible, proper, and accurate. 

Summary 

Special education program evaluation is conducted for the purposes of pro- 
gram improvement, policy analysis, accountability, and public relations. Good pro- 
gram evaluation addresses the program's context, the expectations of people with 
a stake in the program, the program's processes, and products or outcomes of the 
program. 
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A practical approach to evaluation can be based on quantitative methodology 
and structured around the phases of the evaluation process. Specific design elements 
and instrumentation should be based on contextual issues, the purpose of the evalua- 
tion, and stakeholder expectations of the program. Special education administrators 
can determine the relative cost-effectiveness of conducting evaluation, comparing 
the advantages of various levels of outside assistance. When an efficient process 
is employed, evaluation becomes an integral part of administration. 

Successful evaluation requires involvement and ownership in the evaluation 
process by parents and staff. This involvement begins with structuring the evalua- 
tion, and follows through to implementing the management plan. 
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Section 

Three: Qualitative 

Responsive 
Approaches to 
Evaluating Special 
Education Program 
Quality 

fay Edith £. Beatty 
Northeast Regional Resource 
Center 

Burlington, Vermont 

B second paper is written on the 
qualitative approach. This approach uses 
natualistic inquiry to discover the values and expectations held by relevant au- 
diences. The methods used in a qualitative approach evolve throughout the 
process of evaluation. The framework which structures this approach that con- 
siders human and political factors are important. Some comments from ad- 
ministrators throughout the country about the qualitative approach are: 

• We were looking for an evaluation process that allowed us to cap- 
ture the uniqueness of our community and the people in it The 
qualitative method of evaluation provided everyone - directors, 
teachers, and parents -with an easy way of conceptualizing our 
program. 

• This method was selected because it could cover the whole spec- 
trum of services for handicapped students. 

• The process is ongoing and involves the total staff and community 
in evaluation. 

• It was worth it to us to take more time and use this approach since 
it was a valuable catalyst for participation in our district. 

Introduction 

The story goes that three umpires disagreed about the task of calling balls and 
strikes. The first one said, "I calls them as they is." The second one said, i calls 
them as I sees them." The third and most clever umpire said, They ain't nothin'till 
I calls them" (Simons, 1976; p. 29, in Weick, 1979; p. 1). The umpire who 
correctly asserts "They ain't nothin'till I calls them" rather neatly fingers a key 
element in (evaluating): the important role that people play in creating the en- 
vironments that impose on them. (Evaluations), despite their apparent preoc- 
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cupatlon with facta, numbers, objectivity, concretcness, and accountability, arc 
In (act saturated with subjectivity, abstraction, guesses, making do, invention, 
and arbitrariness . . , lust like the rest of us. Much of whot troubles (evaluations) 
Is of their own making, , , (Wolck, 1979; p, 5), 

A local advisory committee for special education decides to make program 
evaluation the number one priority for the coming school year, Never was a deci- 
sion more easi ! y reached. No one could remember a time when such an important 
agenda item breezed by so comfortably and smoothly. Consensus? Everyone in 
agreement? Probably not. Lets look at what the committee members wore 
celebrating In their minds on their way home that night. 

• This will fit In beautifully with our "Excellence In Schools" effort for 
next year. 

• Our staff will learn some good evaluation skills; this will fit right into 
our Comprehensive System of Personnel. Development (CSPD). 

• We need some impact data for our school boards and legislators, 
preferably related to cost-effectiveness; some justification for all 'the 
money expended and appropriated. 

• This provides a forum to showcase our schools; let everyone see that 
ours are the best In the state. 

• Finally, a chance to collect some hard data as to which teachers are 
really teaching our kids. 

• Let's see if we've really done what we said we were going to do. 

• This activity will bring this committee closer together. 

• This will improve the special education programs in the district. 

• The timeliness of this activity is perfect! My daughter is graduating 
soon and the high school still hasn't dealt with her transition. Itll show 
them that secondary programs have to think beyond graduation. 

• This evaluation will show the effectiveness of our three to five year 
old population initiative; we need to prove that we cannot wait any 
longer to provide programs from birth to five. 

• This is a nice logical next step to compliance monitoring. Now we 
can give the state department, and they, In turn, Washington, some 
data which really show what we're doing for handicapped kids. 

• Great! Well provide a picture of all the handicapped kids underserved 
or yet to be served! 

These members, like members of many existing committees and task forces 
at the local, intermediate, state, regional, and national levels throughout the coun- 
try with similar charges, view program evaluation differently, see multiple purposes 
and uses for such an effort, and assume diverse directions and approaches to take 
In order to evaluate the effectiveness of special education programs. Their "celebra- 
tions" might be seen as a collection of issues and concerns which are on the minds 
of stakeholding audiences in or about or affected by the program (Stake, 1975) 
and must not be ignored in the process of the evaluation. They may even be used 
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as the "advance organizers" or guldeposts for the evaluation, The chairperson of 
this committee has a tough and challenging Job ahead, Planning evaluations re- 
quires a great deal of negotiation, Individuals will be tempted to quickly shop for 
good models, tried and true Instruments, and reputable evaluators before considering 
what they actually want from the effort and what they will do with the data they 
will gather. 

Good evaluation requires that the approach fits the context, In order to achieve 
such a fit, the evaluation must be contextually relevant and responsive, and driven 
by the stakeholders. 

A main point for the practitioner is that evaluators may encounter considerable 
difficulties If their perceptions of the study being undertaken differ from those 
of their clients and audiences. Typically, clients want a politically advantageous 
study performed, while the evaluators want to conduct questions-oriented 
studies, since these allow the evaluator to exploit the methodology In which 
they were trained. Moreover, the audiences usually want values-oriented studies 
that will help them determine the relative merits of competing educational goods 
and services. If evaluators are Ignorant of the likely conflict In purpose, the evalua- 
tion Is probably doomed to failure from the start . . . it Is Imperative to remember 
that iio one type of study consistently Is the best In evaluating education. 

It Is virtually impossible to assess the true worth of any object. Such an achieve- 
ment would require omniscience, Infallibility, and a singularly unquestioned value 
base (Stufflebeam and Webster, 1980; p. 17 and 18). 

Special educators need to continue to develop and implement more fitting 
approaches to evaluation efforts and recognize that there are many questions to 
consider prior to selecting a model, a set of instruments, or an evaluator. 

In this paper, the author will describe some responsive evaluation approaches 
which go beyond the scientific rvradigm, ones that are driven by organizers other 
than goals, and ones which rely more on qualitative methods tuan on quantitative 
ones. The issues special educators are currently facing in program evaluation, are 
discussed and explored. Some newer, more fitting approaches, are presented along 
with a set of questions and cues for how responsive evaluation might fit the local 
context and provide for effective evaluation of special education programs. 

The Local Context 
What are the issues? 

As program evaluation has become a nationally high priority, many agen- 
cies, projects, and professional associations have made it their number one area 
for program assistance. A national task force, consisting of representatives from the 
Office of Special Education Programs (OSEP), OSEP supported projects, the six 
Regional Resource Centers (RRCs), the National Association of State Directors of 
Special Education (NASDSE), the Council of Administrators of Special Education 
(CASE), and consumer groups at large, was formed to identify issues and plan 
ways of using available information to address these issues concerning program 
development and evaluation. 

Staff from CASE and the RRC network have identified a host of issues within 
the context of program evaluation. The following section discusses some of the 
issues most relevant to local and intermediate units. 
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Roles and Relationships. Rideral, state, and local levels have historically 
maintained different roles and responsibilities. The push for compliance Is strong 
from the top down, while cost effectiveness and the school Improvement priorities 
have their roots in local autonomy, Potential collaborative efforts have become 
confused by differences In agency orientation. 

Compliance Monitoring and Quality Evaluation. Until recently, 
special education program evaluation has focused on compliance monitoring; 
federal agency assumptions tell us that an excellent program Is a compliant one. 
At the local level educators recognize continuing responsibilities for compliance, 
but feel the drive to move beyond thinking only about minimal standards, Special 
education has been using the same technology to measure compliance and ef- 
fectiveness, seeing the two along one continuum. It would do well to recognize 
the differences. It is unlikely that the same evaluation approach will fit both 
processes. 

Regular and Special Education Partnerships. Similar considera- 
tions apply to the regular and special education relationships In defining quality 
and effectiveness. What is a tt g< od" program? Are "effective" programs for nonhan- 
dlcapped students "appropriate* for handicapped students? Many local and state 
agencies are attempting collaborative efforts to evaluate and Improve effectiveness, 
but again, are finding the need to identify the human and political factors in do- 
ing so. Some communities have combined special education program evaluation 
and school approval/school Improvement programs; others have decidedly 
separated the two. 

Logistical Concerns. In organizing for such an ambitious task as pro- 
gram evaluation, local developers are quick to point to some logistical issues and 
concerns regarding the identification and appropriation of resources. Gathering 
fiscal, human, political, and technical support for evaluation (e.g., negotiating 
design decisions, training staff to do evaluation and/or identifying outside "ex- 
perts" using results, paying for evaluations) is a potentially discouraging piece of 
the process. 

If one pays attention to the context of the program to be evaluated and the 
stakeholding audiences in and around the programs, these four Issues and con- 
cerns can be addressed. 

The Problem. 

With increased attention to program quality evaluation, there is a high level 
of interest in moving away from rigid and closed-ended methodologies and toward 
more open and descriptive approaches. w We need more than numbers" and Tradi- 
tional evaluations have not provided us with the data we really need" are examples 
of statements from administrators interested in expanding the current technology. 
People seem to like the notion of qualitative approaches, but are resistant, perhaps, 
because they believe that the methodology lacks structure or technology, that it 
is not for the "real world" and is not the status quo, or that it is more expensive 
and tends to yield "soft data." Some current models discuss audience identification 
and qualitative methods, but fall short of implementing the underlying assump- 
tions and embracing the design process of responsive evaluation. Somewhere along 
the line in evaluation, scientific inquiry was the predominant choice and that evalua- 
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tlon was to focus on how well goals and objectives were met, The methodology 
additional^ would need to be valid, reliable, objective, and perhaps even 
generallzable, Significance o{ the findings would depend on percentages worth 
noting, co isensus, and a singular truth In terms of what lb "good" or meritorious. 

The world Isn't that easy to evaluate. The author's experience At the local 
level with handicapped kids, their teachers, and administrators Indicates that life 
Just "ain't like that"; human beings don't necessarily behave scientifically, predictably, 
or as we have controlled them. Things simply aren't as rational as they seem and 
what we need Is evaluation findings that are easily gathered, credible, and useful, 
It's Imperative then to look at the very assumptions, paradigms, approaches, and 
methodology most fitting to address evaluation questions. 

The current practice by practice could be expanded by exploring the present 
and growing state of the art In educational evaluation. In fact, given how far special 
education has come with regard to program devebpment, Ironically, professionals 
are still employing evaluation methods developed in the early seventies. Educating 
handlcApped children In public schools Is fAr more sophlstlcAted todAy thAn It was 
In the early seventies; so Is educational evaluation. 

The author pulls together some of the current li.;»iklng In educational evalua- 
tion and show - that there are paradigms within which to describe programs other 
than the scientific one and there are advance organizers In addition to goals and 
objectives. It's Important to recognize our culture as having pluralistic values, not con- 
sensual ones, that quality is In the eyes of the beholder, and that worth must be 
a partner to merit. Relevance is at least as important as rigor and one ought to 
consider the tradeoffs of each prior to assuming the need for one over the other. 

Qualitative Responsive Evaluation 

What Is it? As stated earlier, qualitative responsive evaluation meets two 
criteria: It fits the context and is driven by the stakeholders'purposes and uses. 
The author has resisted using a title for this approach because it is not a model, 
but rather a way of thinking about evaluation. Many existing models do fit this 
criteria and an eclectic posture can be assumed as long as the fittingness of assump- 
tions and criteria are in place. In describing what qualitative responsive evalua- 
tion is. three themes are discussed: the frameworks which structure the approach, 
values considerations in the process, and relevance or criteria for Judging 
evaluations. 

Frameworks 

There are many design issues evaluation developers need to consider in order 
to achieve the right fit of approach to evaluation effort; all to often many important 
considerations are left undesigned and consequently the evaluation is less likely 
to respond to identified purposes and uses. There is neither time in local 
evaluatorslives nor room in this paper to be theoretically detailed or complete. 
However, three of what the author believes to be critical pieces in designing pro- 
gram evaluation will be highlighted. 

Paradigms. One of the very first steps of good design is to explore the 
basic assumptions and postures individuals make about "getting at truth," or the 
"paradigm" in which the evaluation will be housed. As will be seen later, 
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methodologies and even to some extent, advance organism, can be mined, 
paradigm! cannot, 

Nearly all of special education evaluation has been designed and conducted 
within the scientific or conventional pm« .rjm, Efforts to evaluate student ability, 
performance, or lEPsare usually nornvrderenced, objectives-based, discrepancy- 
bated, or name combination, Program compliance Is usually monitored by some 
form of the Dttcrvponcv Evaluation Model (DEM) (Provus, 1971), measuring 
discrepancies between legal mandates and program status, Many of the early and 
currently operational program evaluation model* have been designed using some 
variation of DEM or Stufflebeam, et al's (1971) Context* Input, Process, Product 
(CIPP) model, a decisions-based approach, Evaluations within the scientific 
paradigm assume a singular reality, objectivity in the evaluator-evaluatee relation- 
ship, a preference for quantitative data, paper and pencil instruments, and a preor- 
dlnate design (Guba & Lincoln, 1981), 

In search of Improved means of evaluating quality and effectiveness, it is 
suggested one "try on" the assumptions and postures of the "naturalistic 
paradigm* (Guba, 1978; Guba & Lincoln, 1981 and others). Within this 
framework, one assumes that there are multiple realities, that the evaluator and 
respondents Interact and depend on one another, qualitative data are preferred, 
the human Is often the instrument, and the design is emergent and continual (Guba 
and Lincoln, 1981), 

Since Provus 1 and Stufflebearns work in 1971, many of the major evalua- 
tion models have shifted to the naturalistic paradigm and will be discussed in the 
next section. 

Advance Organizer*. Another early consideration for the evaluator(s) 
is the question of what will guide the evaluation, or how the evaluation will be 
organized in advance. 

Most early efforts, including DEM, were organized around goals or objec- 
tives (Tyler, 1950; Stake, 1967; Popham, 1975). With the great school improve- 
ment movement in the 1960s, conventional evaluation approaches were found 
to be inadequate to address the evaluation needs of the many ambitious pro- 
grams. Evaluation changed dramatically and the following are only examples of 
the many subsequently developed approaches. 

In 1963, Lee Cronbach introduced decisions as a guidepost, as the CIPP 
model later dU, focusing the evaluation for decisionmakers. 

Michael Scriven, in 1974, still operating within the scientific paradigm, iden- 
tified effects is a new advance organizer in his Goa/-Free evaluation model. He 
questioned the rWcrenriation between intended and unintended results and the 
assumption that meeting only intended results would warrant success. He and 
his teams of "professional evaluators" went in to study effects whether they were 
intended or not against a profile of needs. Scriven also contributed the concepts 
of "needs assessment* and the "formative and summarJve ,, distinction in evaluation. 

Eliot Eisner was the first to break out of the scientific paradigm, beginning 
the challenge to the appropriateness of science for evaluating programs concern- 
ed with human behavior. Eisner's Connoisseurshlp model was the first to employ 
the human being as the instrument using the metaphor of an art critic or an Olym- 
pic judge. 
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Introducing value* and acknowledging pluralism in our society was a signify 
cant step outside o( the scientific paradigm, Robert Stake, who had earlier authored 
a popular oujectlves^based evaluation model developed Weiponsiwe Evaluation 
This model was responsive to the issues and concerns of stakeholdlna audiences 
or relevant publics, and was not preordinately designed, He recognised the tradeoff 
of precise measurement in order to increase the usefulness of the finding* to 
stakeholding audiences, 

Cuba (1978), and Guba and Lincoln (1981), have pulled together a corn* 
prehensive and eclectic model, the Nafwra/isfte flespons/pe Evaluation Mode/ 
(NRE). designed within the Natuallstic paradigm, Their model stretches the respotv 
slveness of Stake's design to respond to human and political factors, qualitative 
or quantitative methods depending on the identified issues, concerns, and repor- 
ting needs of the relevant audiences, 

The adversarial process similar to that used In tin administrative hearing in 
the law is the design of the Judicial Mode/ developed by Robert Wolf In 1974 
and is another example of an issues-organized model within the naturalistic 
paradigm. 

The Illumination Mode/ proposed by Farlett and Hamilton in 1977 is similar 
to the responsive model and Is designed to illuminate the problems and issues 
and focus on information gathering rather than decision making. 

Guba (1978), Guba and Lincoln (1981), and Stufflebeam and Webster 
(1980) provide a wealth of information on the various models and their descrip- 
tors as well as references for the writings of the models'authors. These brief descrip- 
tions have been Included to provide the reader with an overview of the continued 
development and trends in educational evaluation. 

Methodology. Once the assumptions, posture and advance organizers 
are designed, other remaining framework considerations are more easily selected 
and organized. Being responsive to issues and concerns of stakeholders within 
the naturalistic paradigm means that some combination of quantitative data 
(demonstrated by numbers) and qualitative data (portrayed In words) will be 
gathered depending on audiences'reporting needs, with the likelihood of the 
evaluation being primarily qualitative. Evaluating "quality* or "effect v *sV* of 
special education programs seems an appropriate arena for employing method* 
such as interviewing and observation to yield case studies, quality indicator vignet- 
tes, portrayals, and other thick descriptions. 

Evaluators of network television are no longer relying on "h?sd counts* or 
numbers of viewers to determine the success and future of television shows; they 
are using qualitative evaluation of viewsrsVesponses to programming with suc- 
cess and now would not return to relying only on the quantitative. Through inter- 
viewing, they ar" finding out not only how many people are watchi ,g what, but 
also who's watching, why, and when. Advertisers used to assume that "if they watch 
it, they like it." Now, through interviewing viewers, they know more about how 
people use television and how they feel about the shows they watch (Roberts, 
1984). 

There is an analogy to be drawn. The author is choosing to expand assump- 
tions that if child counts are proportionate, Wds are in programs and achieving 
and progressing measurably, then there are quality programs. Those criteria are 



not usually what make or break a program Individual* 1 perception* of how well 
the program ii working, how th« teachers are teaching, and how the kldi feel 
about being In those programi are only example of the qualitative data needed 
to describe and judge program effectiveness. 

Valuta 

Qualitative Responsive Evaluation (alb within the naturalistic paradigm 
Therefore, In "getting n\ truth" ipecial education evaluator* resit! the temptation 
to seek any one "correct reality* that exist* if only we could uncover it Living through 
any presidential election, the 1960s, a jury trial or the advisory committee meeting 
described earlier In this paper, provide proof enough that the world Is a pluralistic 
society with multiple values as well as multiple views of quality, Individuals need 
to respond to the divergent expectations of those commissioning and using evalua* 
lions across at! levels; human and political factors greatly affect the success or failure 
of attempts to evaluate program effectiveness. 

At the local level, as at the state and federal agencies, there are several au- 
diences which carry with them countless issues and concerns which they want pro* 
gram evaluation to address, only some of which are compatible. Nowhere does 
it seem more obvious than In the context of special education. It used to be that 
decision making boards were comprised of those who "knew best " The concept 
of who knows best has become quite broader. What is gUven up Is the pursuit of 
what is best for all kids; there are now advising and governing boards collaboratively 
designed to invite multiplicity and even interference to meet the needs of handicap- 
ped kids who have a host of audiences affected by the programs which serve them. 
Homogenizing the data is not a helpful pursuit and often responds to no one's needs. 
Responding to the Issues, concerns, claims, and values of identified stakeholders, 
even ones not of the same persuasion, predicts greater utility for the evaluation 
results. 

Stakeholders. Beginning the implementation of the evaluation design 
by Identifying those Ind 1 'iduals who have a stake In the evaluation outcomes and 
garnering what their expectations of the purposes and uses are Is simply good 
practice. Using issues and concerns as advance organizers rather than Intended 
program goals or objectives Is also good practice, since goals are likely to have been 
»€ 'gncd by a singular audience and may or may not be adequate or ambitious. 

Addr- '.rig stakeholders at the onset is to be responsive to some of the 
previously identified issues in the context of local program evaluation. If state and 
local agencies recognize one another as having a stake In the other's efforts, then 
developers can work collaboratively for mutual responsiveness rather than in com- 
petition for orientation or conflict of responsibility. What is needed Is a less perfect 
model and a move tiexllle approach. 

Similarly, Issues of definir 3 "improvement" or Effectiveness" for regular and 
special educator, -'King evaluation precf: m :o measuring compliance and 
describing or ju'ijiiiq t io?'ty tan ba *ddre<?! * i by delineating the issues and con- 
cerns O stakeho, v r <n<i responding to them throughout the design. 

Merit and HI/* rib. The -roach to evaluating presented thus far is one 
of describing or judging, rath. han measuring or assessing. Evaluation, or the 
process of valuing, um be . : <*ed at on two planes. We can describe an entity, 
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or in thi# em> a special education program, In two way*, tNi difWnetion can be 
made by looking at merit and value (Scriven, 1978) or "merit and worth" (Guba 
& Lincoln, 1981. pp 39 52) Merit is the intrinsic value a program ha* on it* own, 
free of contextual considerations or applicaton, worth it* the utility lomething hat 
within a context or applied to a kmc** P»-w>«* The concept of merit and worth 
is helpful in our co? vi Vi ^ " notion of replicahility, and the 

compliance and mu**'v M»«4t*j».*iMp 

Ucal control, or autonomy, critical to community Social moras and 
values differ not only from state to state, hut also from ciry to town, and house hold 
to household Thus, a m*ritorJo*j* m P&lo Altos school* may not be worth 

replicating in South Pari*, Main* 

The author wai chaBenged on making such a statement at a national meeting 
by the question, "Do you meant to say, what's good for handicapped children in 
California ti not good for handicapped children in Maine?* 'Exactly, she thought* 
feeling that the had mad* her point clear, Her colleague continued with his argu 
ment to say that that's not using what we know, That's tike penicillin," he said, lm< 
plying that she was asserting that each state or local agency should wait to discover 
their own defense against bacteria. "NaT she said, "penicillin ts like monitoring (or 
merit); vitamins, apples, or vegetarian diets - deciding what proactive and sup* 
plemental measures to take In addition - Is what we don't want to standardize, 
We want to leave It to the judgment of worth " 

Beyond the standards set by law and policy, local communities like to main* 
tain control of decisions regarding the search of excellence or Improvement relative 
to local values and educational philosophy* 

Stat*! staff from Rhode Island, the smallest state In the union, tell it all when 
they say, "We can leave our office at eight o'clock In the morning, tour each district 
In the ocean state, and be back for lunch. But, within our state we have rural, 
urban, suburban, coastal, Island, wealthy, poor, and otherwise diverse organtza* 
tions." Rhode Island developers chose a self study guide which allowed for local 
flexibility, and predict that no two sltes'program evaluations will look alike. If local 
control and non-generallacabtlity exist In Rhode Island, It's fair to assume that It's 
the case throughout the nation. 

Standards. Special educators are pleased that the standards set forth In 
the Education of the Handicapped Act were developed; unquestionably, the lives 
of handicapped students are qualitatively better today than they were prior to 
1975. They also share a common goal of developing newer and higher standards, 
or quality Indicators, for educating handicapped students In public schools. 

Being overly concerned with standardization, however, may be an Inap- 
propriate pursuit of generalizablllty. Within the naturalistic paradigm, there Is no 
one perfect set of standards which Is consensual and fitting to every context. Stan- 
dards are value-laden; the Ideal of determnlng generic standards for local special 
education programs Is valiant, complicated, and time-consuming. Standards are 
also judgmental; qualitative judgments are made based on values, perceptions, 
and Intuitive Impressions. It seems, therefore, what is needed is u inlmal stan- 
dards, some recommended standards, and possibly some selected standards. This 
would discourage aspiring to describe the perfect special education program quality 
indicators or program evaluation model, or evaluation Instruments to fit all contexts. 
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Relevance 



Mi#h thai m ****** *m #*^m*\ mimwmk *#m vm% «at'««**»* 
\ *4n mmy k*m #M Hp win* *p 

PHwtat Iww wMue m m-*uy W ^wttMWi ikt*i*siwi quaky t» not On 
Pp4 tfandnfti *v4>M4**>" %ten*M ***** t« in awtfytfhtfnwbi* coneri and 
complete, diui ct*iifc4e~ to |wtt<fe*»» o#t ad fCtottbach. JMHU 

In thii lecnon, concern* *>t nyor and relevant* are *ddre**M by describing 
I wo t*t» of criteria I) the procedure %|^vrfk^Uy tWyficki Im addre** the 'mifclwcH 
thine**" frf naturaNif evaluation*, ?} th*? Mandatd* t*y which «%tt tftMxJ evaluations 
can be judged 

TruMworlhinea* 

hofeWwa (4 r*jc* *tfcn fmm 0* *#uju*ret* *n#4 la inmj&i* ^0nt u^jMuwi* 
of *miwm#* i>> the iiyih#ni!*% k4 On iHlofmatk*) fttt*vid«d m4 On inter 
pf#ia<>am that are H^m h H*** <m em t*ti wtntim th# m'<*m*&oa 
And inierpretatiuttt correct? Whtfiftn* tin wfe^matipn ha* purely l***i 
im# or ml^tf Nmii meaning in ma**v %tfu**Mt*? Whether « writ* lou* ul 
con*i»t#nfly? Whethef ihe mt#f|»re!*f¥m# are h** Utm the pamcuia* \mm 
of On iwjuiref? [Gob* fk LwhoH p 8?) 

Lacal evaluation* don't often "glow* with procedure* l# insure validity and 
reliability, In fact, many admittedly are of the "quick <*nd duly* vawty A**ummg 
that individuate concerned with evaluating are t '-oncemed with correctly evaluating* 
the scientific end naturetotic desenpfcm of bar a****** of *uitw«*th*m a* suifcned 
by Guba {1981; ^ HO) are romped 

Aspect Scientific Tettn Naturalistic Term 

truth value internal validity credibibfy 

applicability external validity, transferability 



Each aspect l* restated to fit this context 

CrtdlbHtty - "How do w« inmm that ih« valuatton finding* ar« cmitbW to 
th« identified audience member* particularly the uienT* 

Flttingnese - *How do we Judge the "worth* of other evaluation ttraiegm 
or special education programs to our context, or our to other contexts?" 

Dependability and Confidentiality - "How do we 'audit* both that we used 
sound design and procedure and that we gathered data In ways that other ac- 
complished evaluators would do so?* 

Several activities for these procedures are detailed in the literature; what is 
important here Is that qualitative data do not need to be V>ft" or tacking in the 
standards of rigor. Trustworthiness criteria are more "relevant* to real world evalua- 
tions outside of the laboratory. 

Good Evaluation. The Joint Committee on Standards for Educational Evaluation 
was formed representing aB of the audiences concerned with educational evalua- 
tion to develop Standards for Evaluations of Educational Programs, facets, and 



consistency 
neutrality 



or genet ahiaMity 

reliability 

objectivity 



dependability 
confirmabdtty 
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Materials, an impressive document now considered a primer for good educational 
evaluation. The Jolm Committee was guided by the assumption . , . that sound 
evaluation can promote the understanding and Improvement of education, while 
faulty evaluation can impair It," (Stufflebeam, et al, 1981; p. 5). Evaluations ought 
to be useful, feasible, proper, and accurate according to the Joint Committee 
members who spent years piecing together these guiding principles. Demonstrating 
that qualitative responsive evaluation has grown to fruition, the standards include 
audience Identification, evaluational Interpretation, political viability, human Interac- 
tions, and analysis of qualitative Information as examples of criteria by which evalua- 
tions can be judged. 

Implementing Qualitative 
Responsive Evaluations 

How do you do it? 

Since qualitative responsive evaluation approaches are just beginning to find 
their way into the special education arena, and since local Implemented responses 
to the assumptions and postures are likely to be, "How do we do it?" The introduc- 
tory vignette is revisited to portray the implementation considerations. 

In moving toward fitting evaluation approaches to their local efforts, the local 
advisory committee should address the following questions and cues in their 
negotiations: 

1. Why Evaluate? 

• Decide what purposes and uses are feasible to address. 

• . . .what they will do with all the findings? 

2. What will be evaluated? 

• Responsible members must put boundaries on the effort. 

• Decide scope and limits. 

3. What are the advance organizers? 

• Check the assumptions members are making both about evaluation 
?nd about the special education programs. 

• Consider organizing the evaluation around the issues and concerns 
of stakeholding audiences such as those listed in the introduction. 

4. Who are the relevant audiences? 

• Identify those "who care." 

• Look at those who "plan," those who "use," those who "provide," those 
who "benefit" from, and those who "suffer" from the programs. 

5. What values do they hold? 

• Seek pluralistic perceptions 

• Invite interference and accept conflict early on; save disappointment 
or disaster later. 

6. By what standards should programs be judged? 

• Consider merit (inherent value). 

• Don't forget worth (contextual importance). 
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7. Design and Implement when? 

• Continually design. 

• Begin immediately; don't stop implementing until the program does 
or until you need to furnish a report. 

8. Evaluate where? 

• Conduct evaluation in the natural setting of the program. 

• . . .in the community or the "local context." 

9. Evaluate how? 

• Respond to the expressed needs of the stakeholders. (If they want 
statistics - give them statistics; if they request video-tapes - video- 
tape. . .within the resources of the evaluation) 

• Use as many methods as are available to be responsive and relevant 
(e.g., interviews, observation, content analysis of documents and 
records). 

• Use human instruments! 

10. What to do with the findings? 

• Look for recurring themes. 

• Accept non-homogenized data. If 85% of the respondents make one 
judgment, explore why 15% have a different opinion. 

11. How to report and advertise results? 

• Revisit the issues and concerns and reporting needs of the identified 
audiences. 

• Report feedback continually in various forms and amounts at various 
times (narrative and oral vignettes, portrayals, case studies, technical 
reports, executive summaries). 

12. How to achieve trustworthy data? 

• Spot check information with stakeholders. 

• Use an evaluation team to "triangulate" both good procedures and 
correct findings. 

Summary 

As local, state, and federal audiences continue their quest for better program 
evaluations, requesting "more than numbers" and asking "who cares" are two pro- 
mising steps to finding more contextually fitting approaches. 

Grownups love figures. When you tell them that you have made a new friend, 
they never ask you any questions about essential matters. They never say to 
you, "What does his voice sound like? What games does he love best? Does 
he collect butterflies?" Instead they demand: "How old is he? How many brothers 
has he? How much does he weigh? How much money does his father make?" 
Only from these figures do they think they have learned anything about him. 
If you were to say to the grown-ups: "I saw a beautiful home made of rosy 
brick, with geraniums on the windows and doves on the roof," they would not 
be able to get any idea of that house at all. You would have to say to them: 
"I saw a house that cost $20,000." Then they would exclaim, "Oh, what a petty 
house that is! (Saint Exupery, 1943), 
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Approaches to 
Local Special 
Education Program 
Evaluation 
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The special education administrator 
becomes involved in program evaluation for a variefy of reasons. The administrator 
must \ake the decision based on the best fit with his or her own orientation, the 
stakeholder needs, and the LEA program context, The next section is a review 
of the similarities and differences in the qualitative and quantitative approaches. 

Introduction 

The local special education administrator has no doubt found the previous 
two papers by Horvath and Beatty of considerable interest. The authors defined 
their intentions early in their papers. Horvath stated that his approach was eclectic: 

Thi:; article advocates a specific approach to special education program evaluation 
for special education administrators, and specific strategies for employing the 
recommended approach. The approach is essentially quantitative, combining 
several of the most frequently employed design elements as they apply to special 
education, A major feature of the approach is that various qualitative 
methodologies can added, to complement the basic model. These alternative 
methodologies are described in the companion article within this issues paper, 
(p. 6) 

Beatty told the reader that her approach operated on a different set of assump- 
tions than the "scientific paradigm": 

In this paper, I will describe some responsive evaluation approaches which go 
beyond the scientific paradigm, ones that are driven by organizers other than 
goals, and ones which rely more on qualitative methods than on quantitative 
ones. HI discuss the issues we're currently facing in program evaluation, explore 



Four: 



45 



some newer, more fitting approaches, and will present a set of questions and 
cues for how responsive evaluation might fit the local content and provide for 
effective evaluation of special education programs. 

The authors have both done an excellent job of addressing their stated pur- 
poses, They have clarified their perceptions of the framework in which evaluation 
takes place and have put forth convincing arguments for their approaches. But the 
local special education administrator reading these papers probably has additional 
questions. For example, the administrator may be asking; "What are the utilities 
of each of the models?" tt Are there clear choices between the two?" "Can the assump- 
tions of each be violated in implementing some sort of an eclectic approach com- 
bining the two?" This paper is designed to clarify the similarities and differences 
between these papers and to draw some conclusions and implications for local 
special education administrators. 



Overview 

Reviewer's Approach. This review is grounded in the assumption im- 
plicit in the Beatty paper and explicit in the Horvath paper that local special educa- 
tion program evaluation is both necessary and feasible. The reviewers background 
and experience has focused on pragmatic evaluations and technical assistance 
to state and local special education agencies. Therefore, this review attempts to 
avoid ideological distinctions between the papers when the ideologies have no 
practical implications. The focus is on helping the special education administrator 
make more informed decisions about program evaluation. 

General Impressions. Two general impressions immediately strike the 
reader when comparing these papers. First, it is clear that the authors have ad- 
dressed their task at slightly different levels. Horvath describes in detail his assump- 
tions about evaluation realities and the steps necessary to implement his approach. 
Beatty, on the other hand, spends most of her paper defining her general orien- 
tation and attempting to clarify her perception of the contexts in which evalua- 
tion takes place. This difference in level of detail makes it somewhat difficult to 
contrast the approaches and requires that the contrasts be made in relation to 
generic approaches rather than specific steps. The requirement to make generic 
comparisons may be an advantage because the second impression is that the 
authors struggled in vain to stay within a single methodology. Originally Horvath 
had been asked to advocate "quantitative" methods and Beatty had been asked 
to advocate "qualitative" methods. Thankfully, neither presented a pure method' 
on the far end of this one continuum. Instead, their papers present to the reader 
a rich set of alternatives along many continua. The mixture of methods described 
in each paper is more realistic than a single dimensional approach. As Miles and 
Huberman (1984), state: "The history of research in many fields shows shifts from 
'either-or* to Tx>th-and' formulations." Both papers represent compromises and eclectic 
approaches rather than single paradigms, models or theories. We might better speak 
of their papers as what Stake calls "persuasions" rather than 'models" (1981) . Because 
the terms "paradigm" "model", "theory", and perhaps even "persuasion" are so en- 
cumbered with non-essential connotations, this review will use the term "approach" 
to refer to the content of each paper. 
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Topics for Comparisons, Ton continue have been detected along which 
the two approaches may be compared. The location of each approach of how 
the authors (Beatty and Horvath) would answer the following ten questions: 

1. Who should make decisions about the evaluation? 

2. To what extent is it important that the evaluation meet criteria for 
"good" evaluations? 

3. To what extent should the pva'-jatlon focus on quality versus 
compliance? 

4. Should the evaluation collect, analyze, and report numbers or words? 

5. Should the evaluation lead to specific judgements or to general 
enllghtment on issues and concern*.? 

6. Can we know in advance how the evaluation will be conducted? 

7. Do "stakeholders" have to agree? 

8. Who can conduct the evaluation? 

9. To what extent can existing data be used? 

10. Should the evaluation focus on processes or outcomes? 



Figure 1 

Comparisons of The Two Approaches on Ten Continua 

H Approach proposed by Horvath 
B « Approach proposed by Beatty 

H--B- Stakeholder Driven 

_HB "Rigor" is not Critical 



1. 

2. 



6. 
7. 



Evaluator Driven 

Must Meet Critc :ia for 
"Good" Evaluations 

Emphasis on "Qur.Iity" 



Uses and Rej 
Numbers 

Focuses on Goals 
and Decisions 

Design Is Preordinate 

Stakeholder Consensus 



Needed 

8. Conducted By Internal 
Staff 

9. Uses Existing Data 

10. Focuses on Outcomes 
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-B~ 



H B- 

_.H B-- 

H? B?~- 

H?B? 
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Emphasis on 
Compliance 

Uses and Reports 
Words 

Fetuses on Issues 
and Concerns 

Design Is Emergent 

Stakeholder 

Pluralism 

Incorporated 

Uses Consultants 

-Gathers New Data 

Focuses on 
Processes 



The remainder of thin paper la intended to ihow how ihe author* might amwer 
the first three or four questions in similar ways, the next three or four questions 
In contrasting ways and to predict how the last three questions might be answered 
based on what appears to bo limited Information In the papers. Implications for 
administrators are presented relative to each continuum. 

Areas of Agreement 

The two approaches appear to be in agreement regarding at least three of 
the continue and are not far apart on the fourth. Both approaches require active 
stakeholder Involvement and valid and reliable processes. They both focus on quality 
rather than compliance issues. Finally, they both use a mixture of approaches In- 
cluding the use of numbers and words, 

Continuum *1: Stakeholder Involvement 

The "Stakeholders" in an evaluation are those whom Bcatty defines as those 

who: 

-care about the program; 
-plan the program; 

- use the program; 
-provide the program; 

- benefit from the program; and 

- suffer from the program. 

Both authors indicate that a broad-besed group of such individuals should 
make decisions regarding the evaluation r hey both indicate that stakeholder in- 
volvement is not simply a first step but is - 1 ongoing part of the evaluation process. 

Implications, The administrator selecting either of these approaches must 
be aware that ownership has to Se shared in order for the evaluation to be effec- 
tive. Neither the administrator nor anv individual using either of these approaches 
to evaluation (the tt cvaluator u ) can rr<aiv ( tain control over the entire process and 
expect the resultant evaluation to ba \ Levant and useful for causing change. The 
administrator interested in exploring <*keholder-based evaluations in more detail 
is referred to Bryk (1983). 

Continuum #2: Meeting Criteria for "Good" 
Evaluations 

Standards for Evaluations. Beatty devotes considerable space to the 
issue of judging her evaluation approach whereas Horvath devotes only limited 
space to this issue. However, both discuss the use of "multiple data sources" and 
"triangulating" as ways of creating evaluations that are both feasible and credible 
or valid. Cost is an obvious concern. Horvath spends two paragraphs discussing 
evaluation costs (p. 34). They both reference Standards for Evaluations of Educa- 
tional Programs, Projects and Materials developed by the Joint Committee on 
Standards for Educational Evaluation (1981). Both call for evaluations that are 
useful, feasible, proper, and accurate. 

Cautions. Both authors raise questions about paying so much attention to 
factors to ensure that the Standards for Evaluations are met that the intent of the 
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evaluation i§ loit, Beatty cautioned the reader that "relevance Is at least as Impor- 
tant ai rigor" Horvath suggests that typical measurement theory would be inap^ 
propriately applied to unique local evaluations. Beatty lays that evaluation data 
must be easily gathered, credible, and useful 

Implications Regarding Truitworthtneee. The administrator can- 
not select either approach expecting that it Is an "easy" way of addressing pio- 
gram evaluation, Both tequir* structured, systematic, and replicable procedures 
in order that the resultant information Is "trustworthy" 

Implications Regarding Costa. Both are concerned with costs. There 
are at least three types of costs that must be considered in conducting an evalua- 
tion. The first is the amount contracted for or expended out of pocket to conduct 
the evaluation beyond day to day administrative costs. This would include costs 
for consultants, task force meetings, questionnaire pioduction, and mailing and 
report production. Second is the amount for hidden costs such as staff time, 
overhead, and "ther costs that would be difficult to track in an internal evalua- 
tion. Finally, there may be a "cost" in terms of side effects. Negative side effects 
could include such things as loss of instruction time, a hostile atmosphere during 
an evaluation process, public attention to negative evaluation report, about special 
education programs that may result in budget reductions, etc. The three types 
of costs should not always be isolated as evaluation costs. If the evaluation is an 
integral part of the management practice of the administrator, the cost cannot 
be separated from the decision making process that should be a normal part of 
each administrator's activities. Administrators that gather and use evaluation in- 
formation for program improvement do not object to the.se costs. Those who feel 
that evaluations are forced upon them and that the evaluation results are not in- 
tended for their benefit, will usually find evaluation costs exorbitant *nd will re- 
sent the evaluation process. 

Continuum *3: Emphasis on Quality 

"Quality" Is Distinct frrm "Compliance". Both authors agree that 
their approaches emphasize the evaluation of "quality" and imply that their ap- 
proaches do more than monitor for compliance with federal and state law. Hor- 
vath lament; the fact that many states have tried to extend their monitoring systems 
to serve as systems for evaluating quality. He states: 

A review of several of these extended monitoring systems reveais significant 
problems. The focus of the evaluation tends to be compliance issues and the 
quality of the compliance process. This n different focus from an evaluation 
that Tr easures the quality of the educational process. Extended compliance 
systems generally do not address outcomes (p. 21) 

Beatty agrees. She states that u We tova been using the same technology to mease 
compliance and effed'veness, seeing the two along one continuum. We would do 
iw?ll to recognize their differences. It is unlikely that the same evaluation approach 
M fit both processes.*(p. 48) 

Another View. Olsen has addressed this issue in a number of papers 
(1979, 1984 a,b,c) and has taken an opposing view. He proposes that "monitor- 
ing* is one process of evaluation and that the ct':fe:«:nce relates to the types of 
program standard against which a program is k g judged. Experienced local 
special education administrators have observed six years of monitoring relative 
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to the Education of the Handicapped Act, Local ndmlnlitretori who have hud 
no training in program evaluation may find It difficult to define differences be* 
tween monitoring and evaluation, In this author 1 ! view, what li now perceived 
to be a standard of "quality" may at sometime become a compliance standard 
If It Is supported In research and In public opinion and th v t becomes law or regula- 
tion, Therefore, the difference relates to timing. What currently appears In federal 
and state low consists of extensive procedural requirements, If the laws were revised 
to Include outcome statements, then "monitoring for compliance" could be equated 
with what some people perceive to be evaluation of "quality", 

Implications* The local special education administrator may be unable 
to find authoritative sources outside the school district, l.e. state and federal regula- 
tion*, around which to design the evaluation. Instead, It will be necessary to de- 
pend on the stakeholders to define "quality" when using either of these approaches, 

Continuum *4: Numbers and Words 

Initial Differences, It Is on this continuum that the two authors begin 
to diverge, Although both authors agree that evaluations may involve the use of 
both numbers and wo-ds, they have different levels of emphasis. Beatty suggests 
that qi itative data ar y preferred. The Joint Committee on Standards for educa- 
tional evaluation defines qualitative Information as "Facts and claims presented 
In narrative, not numer'< al, form" (p. 155). Horvath cautions readers that y (the) 
terms quantitative and qualitative, for example, both describe methods that at- 
tempt to measure quality. The Joint Committee on Standards for Educational 
Evaluation define quantitative information as facts and claims that are represented 
by numbers" (p. 155). Horvath's methods clearly lean more toward the use of 
quantitative information. For example, he refers to analyzing data using "main- 
frame statistical packages" and to reporting "percents". 

Not Entirely. The authors are not dogmatic regarding then positions, Hor- 
vath cautio n that "undue emphasis on statistics Is usually somewhat harmful for 
program improvement studies, because the focus too easily becomes the detail 
of the numbers rather than how to improve the program" (p. 30). He also sug- 
gests that *b good deal of anecdotal information, planned and incidental, is usually 
collected during the process". Horvath is not at the far end of the quantitative 
- qualitative continuum. His method is a mixture of quantitative and qualitative 
methodology. Likewise, Beatty suggests that a variety of methods be used, in- 
cluding the gathering of quantitative data, depending on the audiences'needs. 
Both authors allude to the difficulties in analyzing qualitative data but neither 
discusses this difficult task in detail. 

Implications. Neither approach may be categorized simply by making 
a decision about whether it uses numbers or words. When conducting a study under 
either approach one might be using a checklist, a survey form, an interview pro- 
tocol or other similar methods. Both authors would say that the more important 
decision relates to how the information will be used and how the evaluation is 
focused. The newer qualitative methodology requires a different set of skills and 
an administrator may wish to read the 7 page overview in How To Evaluate Educa- 
tion Programs (January 1985) to get a sense of the process and some additional 
references. 
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Areas In Which They Differ 

White the two approach** are very similar in term* of who makes deelilont, 
the extent to which the evaluation muit be trustworthy and the emphasis on qualh 
ty, and the approaches that differ only In terms of degree regarding the use of 
qualitative and quantitative methods, th<*y differ greatly In three other areas. There 
are the areas of focus, the extent 'o which one may know how the evaluation 
will be conducted In advance and *he extent of stakeholder consensus needed, 
These differences are orientation Imu*s that tend to drive the whole evaluation, 
Beatty (and Egon Guba) would prou ily say that one cannot compromise on the 
three Issues listed below, that any compromise would represent hypocrisy and would 
represent an approach inconsistent with their philosophy On the other hand, Hor- 
vath might t* more likely to sa; that one may bo a lectlc regarding these Issues, 
The approach described In his paper (alls on the f.ir <>nd of each of the three con* 
tinua but In practice his orientation is quite eclectic. 

Contlnuum'S: Issues ard Concerns vs. Decisions 

A Clsar Difference. The On difference relates to whether or not the 
administrator should cant about having answers to specific questions wh*n the 
evaluation is done. Beatty's approach is analogous to "turning the light on w . The 
purpose Is to Illuminate tl > issues and concerns, to define the various viewpoints, 
and to describe what is curently happening relative to the Issues and concerns, 
She notes that one may have "minimal,* "recommended," and Elected" standards 
but these standards are not the focus of her evaluation. The Horvath approach, 
on the other hand, Is clearly goal oriented (i.e., standard oriented). He Indicates 
that the stakeholders should define the progr? i objectives and to sort these ob* 
jectives into process objectives, product objec a> * d objectives that deal with 
both process and product. The Horvath appro Is to developing a specific 
management plan and to a follow-up evaluatlc ty approach leads to 

u revisit(ing) the issues and concerns and leporting » s i i identified audiences." 

Implications, The administrator >as to work with the stakeholders to 
define the purpose of the evaluation. If th re are clear standards against which 
the stakeholders would like the school distric special education program Judged 
and the standards have been articulated In goal objectives and/cr tasks, the Hor- 
vath approach may be a more direct method c comparing the district to these 
standards. If on the other hand. f hr administrator u 1 the stakeholder group have 
some nagging concerns and Isaac* bi t are r.r* «?r a in what the problems might 
be, the Beatty approach may be more appropriate There are considerations related 
to the other two main differences between the approaches that may have even 
greater implications for the administrator's choices. 

Continuum #6: Knowing Design in Advance 

Pre-ordlnate Deelgn. The authors differ in how they would answer the 
question "Can we know, in advance how the evaluation will be conducted?" ? n 
the Horvath approach, developing the design and instrumentation Is an exten- 
sive step early in the evaluation process. He states that the first order of work 
"consists of determining how each evaluation question should be answered". He 
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call* (or determining proper evaluation approaches for each objective, develop- 
ing specific instrumentation and assignments, developing design mechanics and 
analysis procedures and conducting pilot tests and quality control measures prior 
to collecting any Information. 

Emsrgsnt Deilen. Beatty's approach ii to have the design emerge from 
the data collection process, Design Is continual and builds upon itself, Each finding 
drives the next data collection procedures. In her model the evaluator goes 
back to the stakeholders with information and requests direction for the next step 
of "turning the light on* She calls this a "responsive" evaluation, There is a hint 
of responsive evaluation in the approach proposed by Horvath, His fifth step (p. 
16) Involves collecting additional information if important evaluation questions 
remain unresolved due to collecting conflicting data, However, 90% of what will 
occur under his approach is clear after the first two stages of his evaluation pro* 
cess. Beattyfc approach Is more incremental At any point in time, one might know 
only 20 or 30% of what might occur in the future using the emergent design 
approach proposed by Beatty 

Implications. The inferences of Sis continuum have implications for the 
special education program administrator related to funding and looking at 
unintended effects. The preordinate design proposed by Horvath is mote clearly 
circumscribed and easier to communicate to funding agencies and school boards. 
It is fairly simple tc prepare an evaluation contract that specifies how much infor- 
mation will be collected, on how large a sample, and how a report will be prepared. 
The contract can specify the goals that will be evaluated and the total cost, The 
funding agency and the administrator can control the extent to which the evalua- 
tion is allowed to deviate from the design. An emergent design is not so easily 
circumscribed. When one adopts the phlosophy thet one cannot know in advance 
what he/she should be specifically looking for, the evaluation requires extensive 
trust of those involved in the evaluation. One can exert only limited power and 
control in such circumstances. The risk of looking at sensitive areas and /or going 
beyond the concerns of the majority are much greater under the Beatty approach 
than the Horvath approach. The benefits of using an emergent design would be 
the ability, by definition, to look at "unintended consequences," u side effects," and 
other processes and outcomes of a special education program that may or may 
not be related to the goal structure of that program. 

Continuum #7: Need for Consensus 

The third area where the two approaches clearly differ is in the extent to which 
the stallholders have to agree regarding the purpose and procedures for the evalua- 
tion. This difference relates to focusing on goals and decisions vs. issues and con- 
cerns (Continuum *5), and the issue on stakeholder involvement (Continuum 
Although both approaches have extensive stakeholder involvement, Horvath in- 
dicates that they must agree on the direction the evaluation is taking whereas Beatty 
indicates that agreement is not only unnecessary, it is artificial and leads to arbitrary 
results. 

The Consensual Approach. Horvath suggests working with the 
stakeholders in a subcommittee format to come to consensus on the goals of 
education and the critical aspects that need to be evaluated. His approach in- 
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volves a working subcommittee that report! to the total committee Critical to hi* 
approach it *a common purpose among the stakeholder* in a progam* tp 38) 

The Pluralistic Approach. The Hearty approach begin* with the 
assumption that there are multiple realities and that minority opinion! have equal 
value to the majority opinion, There is no need to come to consensus She states 
what is given up Is the pursuit of what is best for all kids; there are now advising 
and governing boards collaboratively designed to invite multiplicity and even in 
terference to meet the needs of handicapped (students)", She states that individuals 
need to respond to the divergent expectations of those commissioning and using 
evaluations across all levels* (p 30), In her approach, the evaluation must be de- 
signed to explore why the minority differs and whether or not their perception of 
reality might add light to the Issues or concerns of both the majority and minority 

Implications. The local special education administrator must answer the 
questions regarding the degree of consensus needed carefully. The implications 
of using the Beatty approach are that there is no "correct answer* 1 , Early in the 
evaluation process the stakeholders can be surveyed regarding whether or not 
they agree on the purpose, approach, and use of results from the evaluation If 
there Is consensus, the use of the Horvath approach may be most appropriate, 
If there Is disagreement, the administrator and the stakeholder group will be forced 
to decide whether they feel they must use a democratic process that results in 
consensus or whether the pluralism Inherent in any group will be accepted and 
the evaluation will lespond to eveiyone's needs. 

Continua About Which There Is Insufficient 
Information In the Papers 

The positions of the two authors on the seven continua described thus far 
In this review have been fairly clear. f*here are three other continua along which 
the approaches may vary but the papers did not contain sufficiently extensive or 
parallel Information to make comparisons and contrasts. These three Issues are 
perhaps not as Important as the previous seven but represent decisions that must 
be made by the local special education administrator. 

Continuum *8: Infernal vs. External Evaluators 

Issue Insufficiently Addressed. Neither of the authors devoted much 
space to the issue of using internal or external evaluators. Horvath notes that "most 
special education administrators do not have the time to manage a program evalua- 
tion unless prepared materials/methods and consultive support are provided* (p. 
12), suggesting that consultants may be necessary in the initial stages of an evalua- 
tion but that the management of a program evaluation may be possible by the 
special education administrator him or herself. Beatty raises a large number of 
"who* "what* "where", and "why* questions on pages 33 and 34 of her paper 
and suggests throughout that the appropriate instrument for evaluation is a human 
intervener. However, her paper never specifies who that Individual might be. 

A Question of Thist and Training* It is this reviewer's experience that 
evaluations conducted by internal staff involving only qualitative data are con- 
sidered suspect by audiences that prior to the evaluation had expressed concerns 



regarding program operation* Thu*, if the special education supervisor'* ad 
mtnistratof and supervisors or t#nch#fti conducted interviews with parents in oh 
tftifi perception* regarding process and results, the school hoard and/®* pawn) 
groups may question th# result* On the other hind, if there It a fogU level of 
trust between the stakeholder* and the administrator and u 0 special education 
staff have the resauiees and skife to conduct a qualitative evaluation* m h an 
evaluation can meet stakeholder need* Th» i*«ue *\w reiatei to the extent to 
which the local special education adrwnistratw must he trained in program evalua 
tion practice*- Morvath suggests that it i* not reasonable to enpect full time ad 
ministralor* to have detailed knowledge of the evaluation profession hut suggests 
that they will need tome sort of training to learn overall concept* and became 
intelligent consumer of evaluation service* 

Implication* Consultant assistance may he needed regardless of the ap 
proach selected Under the Horvath approach, the itate CASE organisation may 
wish to meet and come to consensu* on a core let of goal* and objective* for 
special education program* CASE could then request that the itate provide con 
luttant assistance to develop *ome genetic evaluation material* relative to tho*e 
goal* and objective* Local special education administrators could then adapt tuch 
a system with minimal consultant assistance r* tfive to local issues In the Beatty 
approach, consultant assistance is needed tl« ^ughout, especially If credibility it 
in question, Responsive and qualitative evaluations of the type proposed by Beatty 
are not replicable across school districts and therefore externa) support is needed 
in each tat* The Illinois State Board of Education included an e*cetkmt chapter 
on "Selecting An E valuator" In Its Handbook /or Evaluation 0/ Specie/ Education 
Effectiving* (1982). The administrator may wish to read that information {si* 
pages) which is both entertaining (e g titles such as ''Why do Some E valuator! 
Talk Technical Gibberish?") and informative (e g , Choosing from Four Styles and 
Competencies of Evaluators). 

Continuum *9; Using Existing Data 

The extent to which the two authors would advocate the use of enisling data 
rather than gathering new data is not clear. LEAs are Involved in a variety of data 
collection efforts including state education agency (SEA), compliance visits, ac- 
creditation, and school approval processes; regional association accreditations; staff 
evaluation systems; vocational education evaluate) v *nd Part B evaluations, In 
most cases, these systems • * In quantitative c Mmerical data and even 
categorical data I*., • WnoV 1 srefore, this rev f *w* *■ wines that existing data 
would be slightly more useful In applying the F * ^^roarh than In Beattyfc 
responsive evaluation approach. 

Implications. In both cases, whether the « ■ ^rton question Is design* 
ed In advance or devebped at emerging stages of the evaluation, the process 
could and should involve reviews of available quantitative and qualitative Infor- 
mation prior to gathering new data. This continuum does not represent a major 
distinction between the approaches. However, the Special Education Programs 
(SEP) of the US. Department of Education has proposed that SEAs (and sup- 
posedly LEAs) collect less new Information and make better use of existing data 
in conducting evaluations. This continuum is proposed In this paper to raise the 
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awareness of local administrators that extensive information already exists and 
should not be ignored in conducting program evaluations, 

Continuum #10: Processes or Outcomes? 

Evaluations may be designed to look at what Stake (1967) calls the "Tran- 
sactions" or processes in a program, may be designed to look at outcomes or im- 
pact, or may be designed to look at both. It isn't clear if the authors differ on this 
continuum. 

Difference in Emphasis? Horvath suggests that both process and pro- 
duct (outcome) objectives should be drafted by the stakeholders and used in the 
subsequent evaluation. He laments the undue emphasis on processes in com 
pliance monitoring and suggests that to conduct evaluations of quality, additional 
emphasis needs to be placed on looking at outcomes. Consistent with the 
philosophy in her approach, Beatty does not preordain the emphasis of the evalua- 
tion but describes how the stakeholders would make such decisions. Because her 
evaluation approach is not based upon standards one would not expect her to 
specify her position on the continuum. Qualitative evaluations are particularly well 
suited for evaluating processes (How to Evaluate Education Programs, 1985) and 
her approach places more emphasis on qualitative than quantitative data. Im- 
plications for Evaluating Outcomes. There has been a general backlash to em- 
phasis on the procedural requirements in federal law. The concern is that the pro- 
cedures may not have any correlates to outcomes. The public and funding agen- 
cies are looking for more emphasis on outcomes and effects. Either of these 
methods are appropriate for looking at outcomes. Using Horvath's approach, one 
would most likely produce percents of people satisfied with their program, numbers 
of students graduating or achieving at certain levels, and numbers of children 
receiving certain types of services. These quantitative data would be supplemented 
by comments and interpretive information. Using Beatty's approach, a report would 
most likely describe a few case studies explaining why or why not a particular 
student or teacher was satisfied, achieving, or happy. Contrasting examples also 
would be presented. 

Implications for Evaluating Processes. If the evaluation is to focus 
on processes, the user of the Horvath approach might be able to describe the 
extent to which a school district school or teacher adhered to a particular process 
defined in a manual or some other reference source. The user of the Beatty ap- 
proach would more likely describe what was occuring regardless of what an authori- 
ty might think should be occuring, i.e., without an external standard. Both models 
could focus on either outcomes or processes. The type of information would be 
similar but the orientations and presentations would be different. 

Summary 

The special education administrator becomes involved in program evalua- 
tion for a variety of reasons. Each administrator comes to the task with a different 
orientation. Some administrators require extensive structure, organization, and ad- 
vance planning before becoming involved. Others are willing to tolerate extensive 
ambiguity, to tt go with the flow" and are open to change and new directions. Exter- 



55 



nal pressures may be unidirectional or conflicting. The components of the special 
education program to be evaluated may have been in existence for a long period 
of time or may just be emerging, All of these factors directly affect the choice of 
an evaluation approach. The administrator must make the decision based on the 
best fit with his or her own orientation, the stakeholder needs, and the LEA pro- 
gram context. 

Contrasts 

This review has pointed out that for the most part, evaluation approaches 
are eclectic and some compromises are possible. The distinctions between the two 
approaches were frequently found to be ones of degree rather than being "either or". 
There are three areas where compromises using the two approaches presented in 
this document are unlikely Continua 5, 6 and 7 in this paper were described as 
areas where first one cannc u focused both on standards or \ .od 
generically on issues or concerns. Secondly, jne cannot have both i v . . > irate 
and an emergent design. Finally, one cannot both require consensus anu accept 
pluralistic perceptions of reality. 

Conclusion 

Beatty and Horvath have presented approaches to local program evaluation 
that can assist the special education administrator in making decisions about his 
or her approach. The administrator must answer the questions posed on page 39 
of this paper for himself or herself. Careful consideration of these ten issues will in- 
crease the likelihc od that the resultant evaluation will be "accurate, proper, useful, 
and feasible". (Joint Committee on Standards for Educational Evaluation, 1981). 
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Theodore E Riggen 
Leonard G Burrello, Chairperson 

e Research Committee and the two out- 
side consultants brainstormed a set of implication statements which were re-worked 
into a series of questions to provide special education administrators with a 
framework to guide the selection of approaches to qualify program evaluation. The 
implications in this section are listed below: 

1. Context - why, purpose, what program? 

2. Philosophy - beliefs - assumptions 

3. Use of evaluation 

4. Trust - credibility and probability of the outcome 

5. Control - "boundedness' 

6. Evaluation questions or proL jents 

7. Power 

8. Internal/external expertise 

9. Time frames and work scope - doability 

10. Evaluation of the evaluation 

11. Budget 

Introduction 

The purpose of this section is to improve the capacity of the special educa- 
tion administrator to make an informed decision regarding the most appropriate 
type of program evaluation to conduct in the administrator's unique educational 
setting. It :s recommended that the special education administrator consider these 
key analytical questions regarding his/her agency. The answers will serve as a guide 
towcird choosing either a qualitative or quantitative approach or a combination 
thereof. For example, if a special education administrator is operating with a local 
school board that is placing demands on him/her for data-based effectiveness out- 
comes, it would likely be politically unwise to conduct a qualitative study. On the 
other hand, if the district special education staff is placing demands on the special 
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education administrator for Increased participative management and representative 
involvement in planning and decisionmaking, the administrator may want to pur- 
sue a qualitative approach, The decision must be based on a comprehensive analysis 
of the needs of the district and the community, It is hoped that the following ques- 
tions guide the administrator In this decision-making process. 

The questions are organized under the eleven headings noted above. The 
side headings provide a means to record your own response to each of the ques- 
tions the Research Committee poses. 



Self Study Guide 

1. CONTEXT 

• Why is the evaluation being conducted? note* 

• What is the expected outcome of the evaluation? 

• Who are the u key stakeholders" In this evaluation? internal external 

• What is the power base from which the program 

head Is operating? notes 

• Who are the key power people? Are they different 
from some of the key stakeholders? notes 



2. PHILOSOPHY7BELIEFS 

Is there h written problem beliefs and purpose of 

the evaluation? yes no 

Is there a statement of beliefs for the evaluation 
approach being contemplated? yes no 

Is that philosophy based on the stated philosophy 

of the school district or community? yes no 

Is the evaluator's philosophy consistent with that 

of the stakeholder? yes no somewhat 



3. USE OF EVALUATION 

Is the evaluation to be used as a planning tool? yes no somewhat 

Who will use the evaluation to further an 
understanding of the present conditions of the 
program? notes 

Who will use the evaluation to inform others? notes 



4. TRUST 

Is the program head trusted by the stake holders? yes no somewhat 

What factors will increase the probability that the 
evaluation will be perceived as valuable? notes 
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Is the evaluation being conducted to confirm a 

predetermined outcome or decision? yes no somewhat 



5. CONTROL/"BOUNDEDNESS" 

How "controlled" "bounded", or "predesigned" 
should the evaluation be? 

Are standards and outcome measures for the pro- 



gram already clearly stated? 

Are there preexisting contracts that will dictate the 
evaluation design? 

How flexible are the stakeholders in accom- 
modating changes in the evaluation plan? 

Are the evaluation questions already established 
by an external force such as the board, state re- 
quirements, federal guidelines? 

6. PROBLEM STATEMENTS 

Are the problem statements clearly stated? 

Will the answers to the evaluation questions pro- 
vide the needed information? 



minimally very 
yes no somewhat 
yes no somewhat 
minimally very 

yes no somewhat 

yes no somewhat 
yes no somewhat 



Is the system open to the program changes? yes no somewhat 

7. INTERNAL/EXTERNAL EXPERTISE 

Do any staff members have the expertise to con- 
duct the evaluation? yes no somewhat 

Is there an evaluation unit in the district that can 

provide technical assistance or staff to conduct the 

evaluation? yes no somewhat 

If the evaluation is conducted internally, will the 

results have credibility? yes no somewhat 

How will the staff react to an external evaluation? positively negatively 

Will the evaluation be conducted by a member(s) 
of the staff (internal) or by an external person (s), 

or by a combination (external, internal)? internal external combo 



8. TIMEFRAMES 

What timeframes and constraints exist for the 
evaluation? 

Are the timeframes affected by budgetary 
constraints? 



note 

yes no somewhat 
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Does the evaluation process accommodate the 
school calendar? 

What logical factors such as locating 
stakeholders" should be considered? 

At each step of the evaluation process, have the 
training needs of the internal evaluation team 

bcr ' ' '? 



need? 



ific plan to address each training 



yes no somewhat 
note 

yes no somewhat 
yes no somewhat 



yes no 



yes no 



9. EVALUATION OF THE EVALUATION 

Is there a written plan, accepted by stakeholders, 
whereby the evaluation effort will be monitored? 

Is there a written plan, accepted by stakeholders, 
by which the evaluation effort (process and out- 
come) will be evaluated? 

In the overall plan for evaluation, are there 
specific points which allow reprogrammlng as 
necessary? 

Is the plan understandable and concise? 

Are the recommendations practical? 

Are recommendations indepenent or interdepen- 
dent, i.e., Does one depend upon others? 



yes no somewhat 
yes no 

yes no somewhat 



yes no somewhat 



10. BUDGET 

What are the budgetary needs? 

What are the budgetary restraints? 

Does the evaluation process need to be modified 
in ' jht of needs/restraints? 



notes 
notes 

yes no somewhat 
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